| MASAKARI-MONITORS(1) | masakari-monitors | MASAKARI-MONITORS(1) |
NAME
masakari-monitors - masakari-monitors 17.0.1
Contents:
MASAKARI-MONITORS
Monitors for Masakari
Monitors for Masakari provides Virtual Machine High Availability (VMHA) service for OpenStack clouds by automatically detecting the failure events such as VM process down, provisioning process down, and nova-compute host failure. If it detect the events, it sends notifications to the masakari-api.
Original version of Masakari: https://github.com/ntt-sic/masakari
Tokyo Summit Session: https://www.youtube.com/watch?v=BmjNKceW_9A
Monitors for Masakari is distributed under the terms of the Apache License, Version 2.0. The full terms and conditions of this license are detailed in the LICENSE file.
- Free software: Apache license
- Documentation: https://docs.openstack.org/masakari-monitors
- Source: https://git.openstack.org/cgit/openstack/masakari-monitors
- Bugs: https://bugs.launchpad.net/masakari-monitors
Configure masakari-monitors
- 1.
- Clone masakari using:
$ git clone https://github.com/openstack/masakari-monitors.git
- 2.
- Create masakarimonitors directory in /etc/.
- 3.
- Run setup.py from masakari-monitors:
$ sudo python setup.py install
- 4.
- Copy masakarimonitors.conf and process_list.yaml files from masakari-monitors/etc/ to /etc/masakarimonitors folder and make necessary changes to the masakarimonitors.conf and process_list.yaml files. To generate the sample masakarimonitors.conf file, run the following command from the top level of the masakari-monitors directory:
$ tox -egenconfig
- 5.
- To run masakari-processmonitor, masakari-hostmonitor and masakari-instancemonitor simply use following binary:
$ masakari-processmonitor $ masakari-hostmonitor $ masakari-instancemonitor
Features
- •
- TODO
INSTALLATION
At the command line:
$ pip install masakari-monitors
Or, if you have virtualenvwrapper installed:
$ mkvirtualenv masakari-monitors $ pip install masakari-monitors
USAGE
Monitors for Masakari:
masakari-hostmonitor
Monitor Overview
The masakari-hostmonitor provides compute node High Availability for OpenStack clouds by automatically detecting compute nodes failure via monitor driver.
How does it work based on pacemaker & corosync?
- Pacemaker or pacemaker-remote is required to install into compute nodes to form a pacemaker cluster.
- The compute node’s status is depending on the heartbeat between the compute node and the cluster. Once the node lost the heartbeat, masakari-hostmonitor in other nodes will detect the failure and send notifications to masakari-api.
How does it work based on consul?
- •
- If the nodes in the cloud have multiple interfaces to connect to
management network, tenant network or storage network, monitor driver
based on consul is another choice. Consul agents are required to install
into all noedes, which make up multiple consul clusters.
Here is an example to show how to make up one consul cluster.
Consul Usage
Consul overview
Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. Each of these features can be used individually as needed, or they can be used together to build a full service mesh.
The Consul agent is the core process of Consul. The Consul agent maintains membership information, registers services, runs checks, responds to queries, and more.
Consul clients can provide any number of health checks, either associated with a given service or with the local node. This information can be used by an operator to monitor cluster health.
Please refer to Consul Agent Overview.
Test Environment
There are three controller nodes and two compute nodes in the test environment. Every node has three network interfaces. The first interface is used for management, with an ip such as ‘192.168.101.*’. The second interface is used to connect to storage, with an ip such as ‘192.168.102.*’. The third interface is used for tenant, with an ip such as ‘192.168.103.*’.
Download Consul
Download Consul package for CentOS. Other OS please refer to Download Consul.
sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo sudo yum -y install Consul
Configure Consul agent
Consul agent must runs on every node. Consul server agent runs on controller nodes, while Consul client agent runs on compute nodes, which makes up one Consul cluster.
The following is an example of a config file for Consul server agent which binds to management interface of the host.
management.json
{
"bind_addr": "192.168.101.1",
"datacenter": "management",
"data_dir": "/tmp/consul_m",
"log_level": "INFO",
"server": true,
"bootstrap_expect": 3,
"node_name": "node01",
"addresses": {
"http": "192.168.101.1"
},
"ports": {
"http": 8500,
"serf_lan": 8501
},
"retry_join": ["192.168.101.1:8501", "192.168.101.2:8501", "192.168.101.3:8501"]
}
The following is an example of a config file for Consul client agent which binds to management interface of the host.
management.json
{
"bind_addr": "192.168.101.4",
"datacenter": "management",
"data_dir": "/tmp/consul_m",
"log_level": "INFO",
"node_name": "node04",
"addresses": {
"http": "192.168.101.4"
},
"ports": {
"http": 8500,
"serf_lan": 8501
},
"retry_join": ["192.168.101.1:8501", "192.168.101.2:8501", "192.168.101.3:8501"]
}
Use the tenant or storage interface ip and ports when config agent in tenant or storage datacenter.
Please refer to Consul Agent Configuration.
Start Consul agent
The Consul agent is started by the following command.
# Consul agent –config-file management.json
Test Consul installation
After all Consul agents installed and started, you can see all nodes in the cluster by the following command.
# Consul members -http-addr=192.168.101.1:8500 Node Address Status Type Build Protocol DC node01 192.168.101.1:8501 alive server 1.10.2 2 management node02 192.168.101.2:8501 alive server 1.10.2 2 management node03 192.168.101.3:8501 alive server 1.10.2 2 management node04 192.168.101.4:8501 alive client 1.10.2 2 management node05 192.168.101.5:8501 alive client 1.10.2 2 management
- •
- The compute node’s status is depending on assembly of multiple interfaces connectivity status, which are retrieved from multiple consul clusters. Then it sends notifition to trigger host failure recovery according to defined HA strategy - host states and the corresponding actions.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the hostmonitor if you choice monitor driver based on pacemaker.
[host] # Driver that hostmonitor uses for monitoring hosts. monitoring_driver = default # Monitoring interval(in seconds) of node status. monitoring_interval = 60 # Do not check whether the host is completely down. # Possible values: # * True: Do not check whether the host is completely down. # * False: Do check whether the host is completely down. # If ipmi RA is not set in pacemaker, this value should be set True. disable_ipmi_check = False # Timeout value(in seconds) of the ipmitool command. ipmi_timeout = 5 # Number of ipmitool command retries. ipmi_retry_max = 3 # Retry interval(in seconds) of the ipmitool command. ipmi_retry_interval = 10 # Only monitor pacemaker-remotes, ignore the status of full cluster # members. restrict_to_remotes = False # Standby time(in seconds) until activate STONITH. stonith_wait = 30 # Timeout value(in seconds) of the tcpdump command when monitors # the corosync communication. tcpdump_timeout = 5 # The name of interface that corosync is using for mutual communication # between hosts. # If there are multiple interfaces, specify them in comma-separated # like 'enp0s3,enp0s8'. # The number of interfaces you specify must be equal to the number of # corosync_multicast_ports values and must be in correct order with # relevant ports in corosync_multicast_ports. corosync_multicast_interfaces = enp0s3,enp0s8 # The port numbers that corosync is using for mutual communication # between hosts. # If there are multiple port numbers, specify them in comma-separated # like '5405,5406'. # The number of port numbers you specify must be equal to the number of # corosync_multicast_interfaces values and must be in correct order with # relevant interfaces in corosync_multicast_interfaces. corosync_multicast_ports = 5405,5406
If you want to use or test monitor driver based on consul, please modify following configuration.
[host] # Driver that hostmonitor uses for monitoring hosts. monitoring_driver = consul [consul] # Addr for local consul agent in management datacenter. # The addr is make up of the agent's bind_addr and http port, # such as '192.168.101.1:8500'. agent_manage = $(CONSUL_MANAGEMENT_ADDR) # Addr for local consul agent in tenant datacenter. agent_tenant = $(CONSUL_TENANT_ADDR) # Addr for local consul agent in storage datacenter. agent_storage = $(CONSUL_STORAGE_ADDR) # Config file for consul health action matrix. matrix_config_file = /etc/masakarimonitors/matrix.yaml
The matrix_config_file shows the HA strategy. Matrix is combined by host health and actions. The ‘health: [x, x, x]’, repreasents assembly status of SEQUENCE. Action, means which actions it will trigger if host health turns into, while ‘recovery’ means it will trigger one host failure recovery workflow. User can define the HA strategy according to the physical environment. For example, if there is just 1 cluster to monitor management network connectivity, the user just need to configurate $(CONSUL_MANAGEMENT_ADDR) in consul section of the hostmontior’ configuration file, and change the HA strategy in /etc/masakarimonitors/matrix.yaml as following:
sequence: ['manage'] matrix:
- health: ['up']
action: []
- health: ['down']
action: ['recovery']
Then the hostmonitor by consul works as same as the hostmonitor by pacemaker.
masakari-instancemonitor
Monitor Overview
The masakari-instancemonitor provides Virtual Machine High Availability for OpenStack clouds by automatically detecting VMs domain events via libvirt. If it detects specific libvirt events, it sends notifications to the masakari-api.
How does it work?
- •
- It runs libvirt event loop in a background thread.
- Invoking libvirt.virEventRegisterDefaultImpl() will register libvirt’s default event loop implementation.
- Invoking libvirt.virEventRunDefaultImpl() will perform one iteration of the libvirt default event loop.
- Invoking conn.domainEventRegisterAny() will register event callbacks against libvirt connection instances. The callbacks registered will be triggered from the execution context of libvirt.virEventRunDefaultImpl(), which will send notifications to the masakari-api.
- •
- It will reconnect to libvirt and reprocess if disconnected.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the monitor.
[libvirt] # Override the default libvirt URI. connection_uri = qemu:///system
masakari-introspectiveinstancemonitor
Monitor Overview
The masakari-introspectiveinstancemonitor provides Virtual Machine HA for OpenStack clouds by automatically detecting the system-level failure events via QEMU Guest Agent. If it detects VM heartbeat failure events, it sends notifications to the masakari-api.
How does it work?
- •
- libvirt and QEMU Guest Agent are used as the underlying protocol for messaging to and from VM.
- The host-side qemu-agent sockets are used to detemine whether VMs are configured with QEMU Guest Agent.
- qemu-guest-ping is used as the monitoring heartbeat.
- •
- For the future release, we can pass through arbitrary guest agent commands to check the health of the applications inside a VM.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the monitor.
[libvirt] # Override the default libvirt URI. connection_uri = qemu:///system [introspectiveinstancemonitor] # Guest monitoring interval of VM status (in seconds). # * The value should not be too low as there should not be false negative # * for reporting QEMU_GUEST_AGENT failures # * VM needs time to do powering-off. # * guest_monitoring_interval should be greater than # * the time to SHUTDOWN VM gracefully. guest_monitoring_interval = 10 # Guest monitoring timeout (in seconds). guest_monitoring_timeout = 2 # Failure threshold before sending notification. guest_monitoring_failure_threshold = 3 # The file path of qemu guest agent sock. qemu_guest_agent_sock_path = \ /var/lib/libvirt/qemu/org\.qemu\.guest_agent\..*\.instance-.*\.sock
masakari-processmonitor
Monitor Overview
The masakari-processmonitor, provides key process High Availability for OpenStack clouds by automatically detecting the process failure. If it detects process failure, it sends notifications to masakari-api.
If your OpenStack service runs in container(pod), this processmonitor will not work as expected. It is recommended not to deploy processmonitor.
How does it work?
- •
- Processes to be monitored should be pre-configured in process_list.yaml file.
Define one process to be monitored as follows:
process_name: [Name of the process as it in 'ps -ef'.] start_command: [Start command of the process.] pre_start_command: [Command which is executed before start_command.] post_start_command: [Command which is executed after start_command.] restart_command: [Restart command of the process.] pre_restart_command: [Command which is executed before restart_command.] post_restart_command: [Command which is executed after restart_command.] run_as_root: [Bool value whether to execute commands as root authority.]
Sample of definitions is shown as follows:
# nova-compute process_name: /usr/local/bin/nova-compute start_command: systemctl start nova-compute pre_start_command: post_start_command: restart_command: systemctl restart nova-compute pre_restart_command: post_restart_command: run_as_root: True
- •
- If masakari-processmonitor detects one process failure, it will try to restart it firstly. After several retries failed, it sends notification to masakari-api.
Related configurations
This section in masakarimonitors.conf shows an example of how to configure the monitor.
[process] # Interval in seconds for checking a process. check_interval = 5 # Number of retries when the failure of restarting a process. restart_retries = 3 # Interval in seconds for restarting a process. restart_interval = 5 # The file path of process list. process_list_path = /etc/masakarimonitors/process_list.yaml
MASAKARI MONITORS CONFIGURATION OPTIONS
The following is an overview of all available configuration options in masakari-monitors. To see sample configuration file, see Masakari Monitors Sample Configuration File.
DEFAULT
Determine if monkey patching should be applied.
Related options:
- •
- monkey_patch_modules: This must have values set for this option to have
any effect
List of modules/decorators to monkey patch.
This option allows you to patch a decorator for all functions in specified modules.
Related options:
- •
- monkey_patch: This must be set to True for this option to have any effect
DEPRECATED VARIATIONS
| Group | Name |
| DEFAULT | host |
Full class name for the Manager for instancemonitor.
Full class name for introspectiveinstancemonitor.
Full class name for the Manager for processmonitor.
Full class name for the Manager for hostmonitor.
If set to true, the logging level will be set to DEBUG instead of the default INFO level.
The name of a logging configuration file. This file is appended to any existing logging configuration files. For details about logging configuration files, see the Python logging module documentation. Note that when logging configuration files are used then all logging configuration is set in the configuration file and other logging configuration options are ignored (for example, log-date-format).
DEPRECATED VARIATIONS
| Group | Name |
| DEFAULT | log-config |
| DEFAULT | log_config |
Defines the format string for %(asctime)s in log records. Default: the value above . This option is ignored if log_config_append is set.
DEPRECATED VARIATIONS
| Group | Name |
| DEFAULT | logfile |
DEPRECATED VARIATIONS
| Group | Name |
| DEFAULT | logdir |
Uses logging handler designed to watch file system. When log file is moved or removed this handler will open a new log file with specified path instantaneously. It makes sense only if log_file option is specified and Linux platform is used. This option is ignored if log_config_append is set.
Use syslog for logging. Existing syslog format is DEPRECATED and will be changed later to honor RFC5424. This option is ignored if log_config_append is set.
Enable journald for logging. If running in a systemd environment you may wish to enable journal support. Doing so will use the journal native protocol which includes structured metadata in addition to log messages.This option is ignored if log_config_append is set.
Syslog facility to receive log lines. This option is ignored if log_config_append is set.
Use JSON formatting for logging. This option is ignored if log_config_append is set.
Log output to standard error. This option is ignored if log_config_append is set.
Log output to Windows Event Log.
WARNING:
- Reason
- Windows support is no longer maintained.
The amount of time before the log files are rotated. This option is ignored unless log_rotation_type is set to “interval”.
- Type
- string
- Default
- days
- Valid Values
- Seconds, Minutes, Hours, Days, Weekday, Midnight
Rotation interval type. The time of the last file change (or the time when the service was started) is used when scheduling the next rotation.
Log file maximum size in MB. This option is ignored if “log_rotation_type” is not set to “size”.
Format string to use for log messages with context. Used by oslo_log.formatters.ContextFormatter
Format string to use for log messages when context is undefined. Used by oslo_log.formatters.ContextFormatter
Additional data to append to log message when logging level for the message is DEBUG. Used by oslo_log.formatters.ContextFormatter
Prefix each line of exception output with this format. Used by oslo_log.formatters.ContextFormatter
Defines the format string for %(user_identity)s that is used in logging_context_format_string. Used by oslo_log.formatters.ContextFormatter
- Type
- list
- Default
- ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO']
List of package logging levels in logger=LEVEL pairs. This option is ignored if log_config_append is set.
The format for an instance that is passed with the log message.
The format for an instance UUID that is passed with the log message.
Log level name used by rate limiting: CRITICAL, ERROR, INFO, WARNING, DEBUG or empty string. Logs with level greater or equal to rate_limit_except_level are not filtered. An empty string means that all levels are filtered.
api
Configuration options for sending notifications.
DEPRECATED VARIATIONS
| Group | Name |
| api | tenant-id |
| api | tenant_id |
DEPRECATED VARIATIONS
| Group | Name |
| api | tenant-name |
| api | tenant_name |
Optional domain ID to use with v3 and v2 parameters. It will be used for both the user and project domain in v3 and ignored in v2 authentication.
Optional domain name to use with v3 API and v2 parameters. It will be used for both the user and project domain in v3 and ignored in v2 authentication.
DEPRECATED VARIATIONS
| Group | Name |
| api | user-name |
| api | user_name |
callback
Trial interval of time of the notification processing is error(in seconds).
consul
cors
Indicate whether this resource may be shared with the domain received in the requests “origin” header. Format: “<protocol>://<host>[:<port>]”, no trailing slash. Example: https://horizon.example.com
Indicate that the actual request can include user credentials
Indicate which headers are safe to expose to the API. Defaults to HTTP Simple Headers.
Indicate which methods can be used during the actual request.
Indicate which header field names may be used during the actual request.
healthcheck
The path to respond to healtcheck requests on.
WARNING:
Show more detailed information as part of the response. Security note: Enabling this option may expose sensitive details about the service being monitored. Be sure to verify that it will not violate your security policies.
Additional backends that can perform health checks and report that information back as part of a request.
A list of network addresses to limit source ip allowed to access healthcheck information. Any request from ip outside of these network addresses are ignored.
Check the presence of a file to determine if an application is running on a port. Used by DisableByFileHealthcheck plugin.
Check the presence of a file based on a port to determine if an application is running on a port. Expects a “port:path” list of strings. Used by DisableByFilesPortsHealthcheck plugin.
host
Monitoring probes to collect before making the decision to send Masakari notification about the node status. If and only if monitoring_samples consecutive reports have the same status, will the Masakari notification be sent.
Trial interval of time of the notification processing is error(in seconds).
Do not check whether the host is completely down.
Possible values:
- True: Do not check whether the host is completely down.
- False: Do check whether the host is completely down.
If ipmi RA is not set in pacemaker, this value should be set True.
Only monitor pacemaker-remotes, ignore the status of full cluster members.
Timeout value(in seconds) of the tcpdump command when monitors the corosync communication.
The name of interface that corosync is using for mutual communication between hosts. If there are multiple interfaces, specify them in comma-separated like ‘enp0s3,enp0s8’. The number of interfaces you specify must be equal to the number of corosync_multicast_ports values and must be in correct order with relevant ports in corosync_multicast_ports.
The port numbers that corosync is using for mutual communication between hosts. If there are multiple port numbers, specify them in comma-separated like ‘5405,5406’. The number of port numbers you specify must be equal to the number of corosync_multicast_interfaces values and must be in correct order with relevant interfaces in corosync_multicast_interfaces.
- Type
- string
- Default
- autodetect
- Valid Values
- autodetect, cluster, remote
Using this option, one can avoid systemd checks that would establish whether this hostmonitor is running alongside Corosync and Pacemaker (the cluster stack) or Pacemaker Remote (the remote stack).
The default (autodetect) ensures backward compatibility and means systemd is used to check the stack.
introspectiveinstancemonitor
Guest monitoring interval of VM status (in seconds). * The value should not be too low as there should not be false negative * for reporting QEMU_GUEST_AGENT failures * VM needs time to do powering-off. * guest_monitoring_interval should be greater than * the time to SHUTDOWN VM gracefully. * e.g. | 565da9ba-3c0c-4087-83ca | iim1 | ACTIVE | powering-off | Running
Failure threshold before sending notification.
- The file path of qemu guest agent sock.
- Please use Python raw string notation as regular expressions.
e.g. r’/var/lib/libvirt/qemu/org.qemu.guest_agent..*.instance-.*.sock’
libvirt
oslo_middleware
DEPRECATED VARIATIONS
| Group | Name |
| DEFAULT | osapi_max_request_body_size |
| DEFAULT | max_request_body_size |
Whether the application is behind a proxy or not. This determines if the middleware should parse the headers or not.
process
Interval between re-sending a notification in processmonitor(in seconds).
The file path of process list.
MASAKARI MONITORS SAMPLE CONFIGURATION FILE
Configure Masakari Monitors by editing /etc/masakarimonitors/masakarimonitors.conf.
No config file is provided with the source code, it will be created during the installation. In case where no configuration file was installed, one can be easily created by running:
tox -e genconfig
To see configuration options available, please refer to Masakari Monitors Configuration Options.
AUTHOR
unknown
COPYRIGHT
2024, OpenStack Foundation
| April 5, 2024 | 17.0.1 |