8 Internal checks

Overview

Internal checks allow to monitor the internal processes of Zabbix. In other words, you can monitor what goes on with Zabbix server or Zabbix proxy.

Internal checks are calculated:

• on Zabbix server - if the host is monitored by server
• on Zabbix proxy - if the host is monitored by proxy

Internal checks are processed by server or proxy regardless of the host maintenance status.

To use this item, choose the Zabbix internal item type.

Performance

Using some internal items may negatively affect performance. These items are:

• zabbix[host,,items]
• zabbix[host,,items_unsupported]
• zabbix[hosts]
• zabbix[items]
• zabbix[items_unsupported]
• zabbix[queue,,]
• zabbix[requiredperformance]
• zabbix[stats,,,queue,,]
• zabbix[triggers]

The System information and Queue frontend sections are also affected.

Supported checks

The item keys are listed without customizable parameters and additional information. Click on the item key to see the full details.

Item key details

• Parameters without angle brackets are mandatory and must be used as is (for example, "host" and "available" in zabbix[host,<type>,available]).
• Parameters with angle brackets < > must be replaced with a valid value. If a parameter has a default value, it can be omitted.
• Values for items and item parameters labeled "not supported on proxy" can only be retrieved if the host is monitored by server. Conversely, values "not supported on server" can only be retrieved if the host is monitored by proxy.

zabbix[boottime]

The startup time of Zabbix server or Zabbix proxy process in seconds.
Return value: Integer.

zabbix[cluster,discovery,nodes]

Discovers the high availability cluster nodes.
Return value: JSON object.

Comments:

• This item can be used in low-level discovery.

zabbix[connector_queue]

The count of values enqueued in the connector queue.
Return value: Integer.

zabbix[discovery_queue]

The count of network checks enqueued in the discovery queue.
Return value: Integer.

zabbix[host,,items]

The number of enabled items (supported and not supported) on the host.
Return value: Integer.

zabbix[host,,items_unsupported]

The number of enabled unsupported items on the host.
Return value: Integer.

zabbix[host,,maintenance]

The current maintenance status of the host.
Return values: 0 - normal state; 1 - maintenance with data collection; 2 - maintenance without data collection.

Comments:

• This item is always processed by Zabbix server regardless of the host location (on server or proxy). The proxy will not receive this item with configuration data.
• The second parameter must be empty and is reserved for future use.

zabbix[host,active_agent,available]

The availability of active agent checks on the host.
Return values: 0 - unknown; 1 - available; 2 - not available.

zabbix[host,discovery,interfaces]

The details of all configured interfaces of the host in Zabbix frontend.
Return value: JSON object.

Comments:

• This item can be used in low-level discovery.
• This item is not supported on Zabbix proxy.

zabbix[host,<type>,available]

The availability of the main interface of a particular type of checks on the host.
Return values: 0 - not available; 1 - available; 2 - unknown.

Parameters:

• type - agent, snmp, ipmi, or jmx.

Comments:

• The item value is calculated according to the configuration parameters regarding host unreachability/unavailability.

zabbix[hosts]

The number of monitored hosts.
Return value: Integer.

zabbix[items]

The number of enabled items (supported and not supported).
Return value: Integer.

zabbix[items_unsupported]

The number of unsupported items.
Return value: Integer.

zabbix[java,,<param>]

The information about Zabbix Java gateway.
Return values: 1 - if <param> is ping; Java gateway version - if <param> is version (for example: "7.2.0").

Parameters:

• param - ping or version.

Comments:

• This item can be used to check Java gateway availability using the nodata() trigger function.
• The second parameter must be empty and is reserved for future use.

zabbix[lld_queue]

The count of values enqueued in the low-level discovery processing queue.
Return value: Integer.

Comments:

• This item can be used to monitor the low-level discovery processing queue length.

zabbix[preprocessing_queue]

The count of values enqueued in the preprocessing queue.
Return value: Integer.

Comments:

• This item can be used to monitor the preprocessing queue length.

zabbix[process,<type>,<mode>,<state>]

The percentage of time a particular Zabbix process or a group of processes (identified by <type> and <mode>) spent in <state>. It is calculated for the last minute only.
Return value: Float.

Parameters:

• type - for server processes: agent poller, alert manager, alert syncer, alerter, availability manager, configuration syncer, configuration syncer worker, connector manager, connector worker, discovery manager, discovery worker, escalator, ha manager, history poller, history syncer, housekeeper, http agent poller, http poller, icmp pinger, ipmi manager, ipmi poller, java poller, lld manager, lld worker, odbc poller, poller, preprocessing manager, preprocessing worker, proxy group manager, proxy poller, self-monitoring, service manager, snmp poller, snmp trapper, task manager, timer, trapper, trigger housekeeper, unreachable poller, vmware collector;
  for proxy processes: agent poller, availability manager, configuration syncer, data sender, discovery manager, discovery worker, history syncer, housekeeper, http agent poller, http poller, icmp pinger, ipmi manager, ipmi poller, java poller, odbc poller, poller, preprocessing manager, preprocessing worker, self-monitoring, snmp poller, snmp trapper, task manager, trapper, unreachable poller, vmware collector;
• mode - avg - average value for all processes of a given type (default);
  count - returns number of forks for a given process type, <state> should not be specified;
  max - maximum value;
  min - minimum value;
  <process number> - process number (between 1 and the number of pre-forked instances; for example, if 4 trappers are running, the value is between 1 and 4);
• state - busy - process is in busy state, for example, the processing request (default);
  idle - process is in idle state doing nothing.

Comments:

• If <mode> is a Zabbix process number that is not running (for example, with 5 pollers running the <mode> is specified to be 6), such an item will turn unsupported.
• Minimum and maximum refers to the usage percentage for a single process. So if in a group of 3 pollers usage percentages per process were 2, 18 and 66, min would return 2 and max would return 66.
• Processes report what they are doing in shared memory and the self-monitoring process summarizes that data each second. State changes (busy/idle) are registered upon change - thus a process that becomes busy registers as such and doesn't change or update the state until it becomes idle. This ensures that even fully hung processes will be correctly registered as 100% busy.
• Currently, "busy" means "not sleeping", but in the future additional states might be introduced - waiting for locks, performing database queries, etc. Note that asynchronous pollers are considered busy if they have reached the limit set by the MaxConcurrentChecksPerPoller server/proxy configuration parameter.
• On Linux and most other systems, resolution is 1/100 of a second.

Examples:

zabbix[process,poller,avg,busy] #the average time of poller processes spent doing something during the last minute
       zabbix[process,"icmp pinger",max,busy] #the maximum time spent doing something by any ICMP pinger process during the last minute
       zabbix[process,"history syncer",2,busy] #the time spent doing something by history syncer number 2 during the last minute
       zabbix[process,trapper,count] #the amount of currently running trapper processes

zabbix[proxy,<name>,<param>]

The information about Zabbix proxy.
Return value: Integer.

Parameters:

• name - the proxy name;
• param - lastaccess - the timestamp of the last heartbeat message received from proxy;
  delay - how long the collected values are unsent; calculated as "proxy delay" + ("current server time" - "proxy lastaccess"), where "proxy delay" is the difference between the current proxy time and the timestamp of the oldest unsent value on proxy.

Comments:

• This item is always processed by Zabbix server regardless of host location (on server or proxy).
• The fuzzytime() function can be used to check the availability of proxy.

Example:

zabbix[proxy,"Germany",lastaccess] #the timestamp of the last heartbeat message received from "Germany" proxy

zabbix[proxy,discovery]

The list of Zabbix proxies with name, mode, encryption, compression, version, last seen, host count, item count, required values per second (vps), version status (current/outdated/unsupported), timeouts by item type, proxy group name (if proxy belongs to group), state (unknown/offline/online).
Return value: JSON object.

zabbix[proxy group,<name>,available]

The number of online proxies in a proxy group.
Return value: Integer.

Parameters:

• name - the proxy group name.

zabbix[proxy group,<name>,pavailable]

The percentage of online proxies in a proxy group.
Return value: Float.

Parameters:

• name - the proxy group name.

zabbix[proxy group,<name>,proxies]

The list of Zabbix proxies in a proxy group with name, mode, encryption, compression, version, last seen, host count, item count, required values per second (vps), version status (current/outdated/unsupported), timeouts, proxy group name, state (unknown/offline/online).
Return value: JSON.

Parameters:

• name - the proxy group name.

zabbix[proxy group,<name>,state]

The state of a proxy group.
Return value: 0 - unknown; 1 - offline; 2 - recovering; 3 - online; 4 - degrading.

Parameters:

• name - the proxy group name.

zabbix[proxy group,discovery]

Returns a list of proxy groups with configuration data and real-time data. Configuration data include the proxy group name, failover delay, and the minimum number of online proxies required. Real-time data include the proxy group state (see comments for details), the number of online proxies, and the percentage of online proxies.
Return value: JSON.

Comments:

• This item does not return groupless proxies.
• If there is an invalid value for "failover_delay" or "min_online", then a special value -1 is reported to indicate that. Invalid values might occur if macros are used for configuration and the macros cannot be expanded to a valid value.
• The proxy group state is reported as an integer: 0 - unknown; 1 - offline; 2 - recovering; 3 - online; 4 - degrading.

Example of return values:

{
           "groups": [
              { "name": "Riga", "failover_delay": 60, "min_online": 1 },
              { "name": "Tokyo", "failover_delay": 60, "min_online": 2 },
              { "name": "Porto Alegre", "failover_delay": 60, "min_online": 3 }
           ],
           "details": {
               "Riga": { "state": 3, "available": 10, "pavailable": 20 },
               "Tokyo": { "state": 3, "available": 10, "pavailable": 20 },
               "Porto Alegre": { "state": 1, "available": 0, "pavailable": 0 }
           }
       }

zabbix[proxy_buffer,buffer,<mode>]

The proxy memory buffer usage statistics.
Return values: Integer (for size); Float (for percentage).

Parameters:

• mode: total - the total size of buffer (can be used to check if memory buffer is enabled);
  free - the size of free buffer;
  pfree - the percentage of free buffer;
  used - the size of used buffer;
  pused - the percentage of used buffer.

Comments:

• Returns a 'Proxy memory buffer is disabled' error when the memory buffer is disabled;
  
• This item is not supported on Zabbix server.

zabbix[proxy_buffer,state,changes]

Returns the number of state changes between disk/memory buffer modes since start.
Return values: Integer; 0 - the memory buffer is disabled.

Comments:

• Frequent state changes indicate that either the memory buffer size or age must be increased.
• If the memory buffer state is monitored infrequently (for example, once a minute) then the buffer might flip its state without it being registered.

zabbix[proxy_buffer,state,current]

Returns the current working state where the new data are being stored.
Return values: 0 - disk; 1 - memory.

Comments:

• "0" is also returned when the memory buffer is disabled.

zabbix[proxy_history]

The number of values in the proxy history table waiting to be sent to the server.
Return values: Integer.

Comments:

• This item is not supported on Zabbix server.

zabbix[queue,<from>,<to>]

The number of monitored items in the queue which are delayed at least by <from> seconds, but less than <to> seconds.
Return value: Integer.

Parameters:

• from - delayed by at least (default is 6 seconds);
• to - delayed by at most (default is infinity).

Comments:

• Time suffixes (s,m,h,d,w) are supported in the parameters.

zabbix[rcache,<cache>,<mode>]

The availability statistics of the Zabbix configuration cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

• cache - buffer;
• mode - total - the total size of buffer;
  free - the size of free buffer;
  pfree - the percentage of free buffer;
  used - the size of used buffer;
  pused - the percentage of used buffer.

zabbix[requiredperformance]

The required performance of Zabbix server or Zabbix proxy, in new values per second expected.
Return value: Float.

Comments:

• Approximately correlates with "Required server performance, new values per second" in Reports → System information.

zabbix[stats,<ip>,<port>]

The internal metrics of a remote Zabbix server or proxy.
Return values: JSON object.

Parameters:

• ip - the IP/DNS/network mask list of servers/proxies to be remotely queried (default is 127.0.0.1);
• port - the port of server/proxy to be remotely queried (default is 10051).

Comments:

• The stats request will only be accepted from the addresses listed in the 'StatsAllowedIP' server/proxy parameter on the target instance.
• A selected set of internal metrics is returned by this item. For details, see Remote monitoring of Zabbix stats.

zabbix[stats,<ip>,<port>,queue,<from>,<to>]

The internal queue metrics (see zabbix[queue,<from>,<to>]) of a remote Zabbix server or proxy.
Return values: JSON object.

Parameters:

• ip - the IP/DNS/network mask list of servers/proxies to be remotely queried (default is 127.0.0.1);
• port - the port of server/proxy to be remotely queried (default is 10051);
• from - delayed by at least (default is 6 seconds);
• to - delayed by at most (default is infinity).

Comments:

• The stats request will only be accepted from the addresses listed in the 'StatsAllowedIP' server/proxy parameter on the target instance.
• A selected set of internal metrics is returned by this item. For details, see Remote monitoring of Zabbix stats.

zabbix[tcache,<cache>,<parameter>]

The effectiveness statistics of the Zabbix trend function cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

• cache - buffer;
• mode - all - total cache requests (default);
  hits - cache hits;
  phits - percentage of cache hits;
  misses - cache misses;
  pmisses - percentage of cache misses;
  items - the number of cached items;
  requests - the number of cached requests;
  pitems - percentage of cached items from cached items + requests. Low percentage most likely means that the cache size can be reduced.

Comments:

• This item is not supported on Zabbix proxy.

zabbix[triggers]

The number of enabled triggers in Zabbix database, with all items enabled on enabled hosts.
Return value: Integer.

Comments:

• This item is not supported on Zabbix proxy.

zabbix[uptime]

The uptime of the Zabbix server or proxy process in seconds.
Return value: Integer.

zabbix[vcache,buffer,<mode>]

The availability statistics of the Zabbix value cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

• mode - total - the total size of buffer;
  free - the size of free buffer;
  pfree - the percentage of free buffer;
  used - the size of used buffer;
  pused - the percentage of used buffer.

Comments:

• This item is not supported on Zabbix proxy.

zabbix[vcache,cache,<parameter>]

The effectiveness statistics of the Zabbix value cache.
Return values: Integer. With the mode parameter returns: 0 - normal mode; 1 - low memory mode.

Parameters:

• parameter - requests - the total number of requests;
  hits - the number of cache hits (history values taken from the cache);
  misses - the number of cache misses (history values taken from the database);
  mode - the value cache operating mode.

Comments:

• Once the low-memory mode has been switched on, the value cache will remain in this state for 24 hours, even if the problem that triggered this mode is resolved sooner.
• You may use this key with the Change per second preprocessing step in order to get values-per-second statistics.
• This item is not supported on Zabbix proxy.

zabbix[version]

The version of Zabbix server or proxy.
Return value: String. For example: 7.2.0.

zabbix[vmware,buffer,<mode>]

The availability statistics of the Zabbix vmware cache.
Return values: Integer (for size); Float (for percentage).

Parameters:

• mode - total - the total size of buffer;
  free - the size of free buffer;
  pfree - the percentage of free buffer;
  used - the size of used buffer;
  pused - the percentage of used buffer.

zabbix[vps,written]

The total number of history values written to database.
Return value: Integer.

zabbix[wcache,<cache>,<mode>]

The statistics and availability of the Zabbix write cache.
Return values: Integer (for number/size); Float (for percentage).

Parameters:

• cache - values, history, index, or trend;
• mode - (with values) all (default) - the total number of values processed by Zabbix server/proxy, except unsupported items (counter);
  float - the number of processed float values (counter);
  uint - the number of processed unsigned integer values (counter);
  str - the number of processed character/string values (counter);
  log - the number of processed log values (counter);
  text - the number of processed text values (counter);
  not supported - the number of times item processing resulted in item becoming unsupported or keeping that state (counter);
  (with history, index, trend cache) pfree (default) - the percentage of free buffer;
  total - the total size of buffer;
  free - the size of free buffer;
  used - the size of used buffer;
  pused - the percentage of used buffer.

Comments:

• Specifying <cache> is mandatory. The trend cache parameter is not supported with Zabbix proxy.
• The history cache is used to store item values. A low number indicates performance problems on the database side.
• The history index cache is used to index the values stored in the history cache.
• After the history cache is filled and then cleared, the history index cache will still keep some data. This behavior is expected and helps the system run more efficiently by avoiding the extra processing required to constantly resize the memory.
• The trend cache stores the aggregate for the current hour for all items that receive data.
• You may use the zabbix[wcache,values] key with the Change per second preprocessing step in order to get values-per-second statistics.