We use Zabbix for montoring of the Openstack clusters. Each cluster/region has a set of Zabbix-proxies that relay monitoring data to the Zabbix server.
Configure discovery rules
The discovery rules help registering the Openstack infrastructure servers to be monitored in Zabbix. We use two rules for each platform; one for servers running the Zabbix-agents and one for the BMC of physical servers (SNMP based).
The following configuration is needed for each discovery rule:
- A descriptive name
- The proxy-server to use for discovery.
- The IP-range to run discovery on.
- A set of check to run for each IP-address.
BMC rule
As the BMC network is highly protected we use the default SNMP community "public". Create discovery checks setting check-type "SNMPv2 agent" and the following OIDs:
- 1.3.6.1.4.1.232.2.2.4.2.0
- 1.3.6.1.4.1.674.10892.2.1.1.4.0
These OID work for HP (ILO) and Dell (iDrac) servers.
Agent/service discovery rule
Add the following checks:
- ICMP ping
- SSH
- TCP:
- Port: 3306
- Port: 5432
- Port: 5672
- Port: 9000
- Port: 11211
- Zabbix agent:
- Key: "system.hostname"
- Key: "system.uname"
- Key: "vfs.file.contents[/sys/class/dmi/id/chassis_vendor]"
- Key: "vfs.file.exists[/etc/apache2/mods-enabled/status.conf,sym]"
Configure discovery actions
We have a number of discovery actions i Zabbix that adds hosts, configures servergroups and links templates as Zabbix agents and other services are discovered.
These actions can be grouped as follows:
- Application/service discovery
- Host (physical/virtual) discovery
- Hardware discovery
- OS discovery
- Location discovery
Application/service discovery
- Apache servers
- Zabbix agent "vfs.file.exists[/etc/apache2/mods-enabled/status.conf,sym]" received value 1, add server to App/Apache Servers host group, link template Apache by Zabbix Agent
- Ceph monitors
- Zabbix agent "system.hostname" contains "cephmon", add server to Ceph/monitors host group, link template NTNU - Ceph by Zabbix agent 2
- Ceph Rados gateways
- Zabbix agent "system.hostname" contains "radosgw", add server to RGWs host group, link template NTNU - Ceph Radosgw
- Neutron DR agents
- Zabbix agent "system.hostname" contains "neutronv", add server to DR-agents host group, link template NTNU - Openstack - BGP DR-Agents
- HAProxy
- TCP port 9000 - Add server to HAProxy host group, link template HAProxy by Zabbix agent
- Memcached
- TCP port 11211 - Add server to Memcached host group, link template Memcached by Zabbix agent 2
- MySQL
- TCP port 3306 - Add server to MySQL host group, link template MySQL by Zabbix agent 2
- PostgreSQL
- TCP port 9000 - Add server to PostgreSQL host group, link template PostgreSQL by Zabbix agent 2
- RabbitMQ
- TCP port 5672 - Add server to RabbitMQ host group, link template RabbitMQ by Zabbix agent
Host discovery
- Physical servers (BMC/ILO/iDrac)
- Virtual servers (Zabbix agent)
Hardware discovery
- GPU discovery
- Physical servers (Zabbix agent)
OS discovery
- Windows OS
Location discovery
- Region discovery (add tag i Zabbix)
Openstack integration configuration
To monitor the Openstack controllplane we need one infrastructure VM (usually named "apimon"-something) in each region to run our Openstack Zabbix template. This VM needs to have the API clients for Openstack and the Zabbix client installed. This is achieved by assigning the server the Puppet role "apimon".
In the "services" project of the Openstack platform there should be a user named "apicheck". Create it as a member of the "services" project if it doesn't exist. Also create a application credential for this user.
Before assigning the "NTNU Openstack" Zabbix-template to the VM, we must configure some Zabbix macro-values that the template needs to be able to access the Openstack APIs. These are summarized in the following table.
| Macro-name | Description | Value | Default |
|---|---|---|---|
| {$OPENSTACK.DATABASE.HOST} | FQDN of the Openstack MySQL cluster | adminapi.<PLATFORM>.ntnu.no | NA |
| {$OPENSTACK.DATABASE.PASSWORD} | Password of the "zabbix_ro"-user on MySQL cluster | Is found in Heira-key ntnuopenstack::zabbix::database::password | NA |
| {$OPENSTACK.DATABASE.USER} | Username to use for MySQL cluster. | Is set by Heira-key ntnuopenstack::zabbix::database::username | zabbix_ro |
| {$OPENSTACK.APPCRED.ID} | App cred ID for the "apicheck" user. | Is found in Heira: profile::zabbix::openstack::appcred::id | NA |
| {$OPENSTACK.APPCRED.SECRET} | App cred secret for the "apicheck" user. | Is found in Heira: profile::zabbix::openstack::appcred::secret | NA |
| {$OPENSTACK.APPCRED.PASSWORD} | The password of the "apicheck" user. | Is found in Heira: profile::zabbix::openstack::apicheck::password | NA |
| {$OPENSTACK.ADMIN.API.FQDN} | adminapi.<PLATFORM>.ntnu.no | NA | |
| {$OPENSTACK.HORIZON.FQDN} | FQDN of Horzion frontend | NA | |
| {$OPENSTACK.KEYSTONE.API.FQDN} | api.<PLATFORM>.ntnu.no | NA | |
| {$OPENSTACK.PUBLIC.API.FQDN} | api.<PLATFORM>.ntnu.no | NA | |
| {$OPENSTACK.SWIFT.API.FQDN} | s3.<PLATFORM>.ntnu.no | NA |
We also check the availability of some webservices from the apimon-server. This is done by the "NTNU - Web" template.
| Macro-name | Description | Value | Default |
|---|---|---|---|
| {$SHIFTLEADER.FQDN} | FQDN for the Shiftleader server for the platform | NA | |
| {$ZABBIX.FQDN} | FQDN for the Zabbix-server for the platform | NA |
In addition to the macros mentioned above, there are some macros that needs to be set specifically for some services.
| Service | Description | Macro-name | Value | Default |
|---|---|---|---|---|
| Apache | {$APACHE.PROCESS_NAME} | apache2 | httpd | |
| HAProxy (adminlb and servicelb servers) | {$HAPROXY.STATS.PATH} | / | NA | |
| {$HAPROXY.STATS.PORT} | 9000 | NA | ||
| MySQL | {$MYSQL.DSN} | /var/run/mysqld/mysqld.sock | NA | |
| {$MYSQL.USER} | zabbix_agent | NA | ||
| RabbitMQ | {$RABBITMQ.API.USER} | Found in Heira profile::rabbitmq::rabbituser | NA | |
| {$RABBITMQ.API.PASSWORD} | Found in Heira profile::rabbitmq::rabbitpass | NA | ||
| PostgreSQL | {$PG.USER} | zbx_monitor | NA | |
| {$PG.PASSWORD} | Found in Heira profile::postgres::zbx_monitor_password | NA |