--- title: "Public Cloud Load Balancer monitoring with Prometheus" description: "Discover the various options available to monitor your Load Balancer" url: https://docs.ovhcloud.com/en/guides/public-cloud/network-services/loadbalancer-monitoring-prometheus lang: en lastUpdated: 2024-03-19 --- # Public Cloud Load Balancer monitoring with Prometheus ## Objective The Public Cloud Load Balancer provides 2 ways to monitor your Load Balancers. You can either query an API to get usage statistics or configure a prometheus listener that can be scraped by your prometheus infrastructure to gather metrics. This guide will discuss those options to monitor your Octavia Load Balancer. ## Requirements - A Load Balancer created. If this is not the case, please check this [guide](/en/guides/public-cloud/network-services/getting-started-load-balancer.md). *** ### OVHcloud Control Panel Access - **Direct link:** Public Cloud Projects - **Navigation path:** Public Cloud > Select your project *** ## Instructions ### Monitoring using CLI The Load Balancer service aggregates these statistics and makes them available via the OpenStack API. Those statistics are available at the Load Balancer or listener level. Load Balancer statistics can be queried using the [OpenStack Client](https://docs.openstack.org/python-openstackclient/latest/). They include metrics for all the listeners configured for the queried Load Balancer. ```bash $ openstack loadbalancer stats show +--------------------+-----------+ | Field | Value | +--------------------+-----------+ | active_connections | 0 | | bytes_in | 2236722 | | bytes_out | 100973832 | | request_errors | 0 | | total_connections | 3606 | +--------------------+-----------+ ``` Individual listener statistics can also be queried using the [OpenStack Client](https://docs.openstack.org/python-openstackclient/latest/). ```bash $ openstack loadbalancer listener stats show +--------------------+-------+ | Field | Value | +--------------------+-------+ | active_connections | 0 | | bytes_in | 89 | | bytes_out | 237 | | request_errors | 0 | | total_connections | 1 | +--------------------+-------+ ``` ### Monitoring with Prometheus To add a Prometheus endpoint on a Public Cloud Load Balancer, create a listener using the `PROMETHEUS` protocol. This will enable the `/metrics` endpoint which by default listens on all the interfaces from your load balancer. **This means that if your load balancer is using a Floating IP, by default the `/metrics` endpoint will be opened to the internet**. To avoid that, we advise setting the `allowed_cidrs` option to protect your metrics, for example with the CIDR of the subnet where the load balancer is spawned. This listener type provides the same features as the "regular" listeners but does not support attaching pools or L7 policies. All metrics will be identified by the Octavia object ID (UUID) of the resources. :::info Currently UDP and SCTP specific metrics are not reported via Prometheus endpoints. ::: #### Create a Prometheus listener via OVHcloud Control Panel Under the `Network ` category, select `Load Balancer `. A page listing the load balancers will be displayed. Select one load balancer by clicking its `name `. The load balancer details page will be displayed. Click the `Listeners ` tab then click on `Add a listener `. In the form: - Fill in the `Name`. - Select `prometheus` as the Protocol. - Choose a port (different from the already existing listener ports of your Load Balancer). The default pool is greyed out because the `prometheus` listener does not need a pool unlike the other types of listeners. ![Create listener in OVHcloud Control Panel](/images/public-cloud/network-services/technical-resources-02-octavia-monitoring-prometheus/create_listener_manager.png) Click `Add ` to trigger the listener creation. The new listener is added to the listeners list. ![Listener list in OVHcloud Control Panel](/images/public-cloud/network-services/technical-resources-02-octavia-monitoring-prometheus/listener_list.png) #### Create a Prometheus listener via OpenStack GUI (Horizon) Log in to Horizon using this [guide](/en/guides/public-cloud/cross-functional/introducing-horizon.md). Click `Network ` then `Load Balancers `. The Load Balancer list is displayed. Click on the load balancer name. The load balancer details page is then displayed. ![Load Balancer details in Horizon](/images/public-cloud/network-services/technical-resources-02-octavia-monitoring-prometheus/horizon_lb_details.png) Click the `Listeners ` tab then `Create Listener `. n the listener creation page, fill in the `Name` and switch the `Protocol` to `PROMETHEUS`. The port will be set to a default value, change it if needed. ![Listener creation in listener](/images/public-cloud/network-services/technical-resources-02-octavia-monitoring-prometheus/horizon_listener_creation.png) #### Create a Prometheus listener via OpenStack CLI To create a Prometheus endpoint on port 9100 for Load Balancer lb1, run the following command: ```bash $ openstack loadbalancer listener create --name stats-listener --protocol PROMETHEUS --protocol-port 9100 lb1 +-----------------------------+--------------------------------------+ | Field | Value | +-----------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2023-11-30T01:44:25 | | default_pool_id | None | | default_tls_container_ref | None | | description | | | id | fb57d764-470a-4b6b-8820-627452f55b96 | | insert_headers | None | | l7policies | | | loadbalancers | b081ed89-f6f8-48cb-a498-5e12705e2cf9 | | name | stats-listener | | operating_status | OFFLINE | | project_id | 4c1caeee063747f8878f007d1a323b2f | | protocol | PROMETHEUS | | protocol_port | 9100 | | provisioning_status | PENDING_CREATE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | None | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | | tls_ciphers | None | | tls_versions | None | | alpn_protocols | None | | tags | | +-----------------------------+--------------------------------------+ ``` Note that you should add the `--allowed-cidr` option in order to filter the listener to a specific network. #### Create a Prometheus listener via Terraform The resource [openstack\_lb\_listener\_v2](https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs/resources/lb_listener_v2) from the OpenStack provider enables to configure a Prometheus listener. The following snippet is extracted from a full example available on [Github](https://github.com/yomovh/tf-at-ovhcloud/tree/main/simple_http_lb_with_prom_grafana), adapt it to your needs. ```python resource "openstack_lb_listener_v2" "prom_listener" { protocol = "PROMETHEUS" protocol_port = 9100 loadbalancer_id = openstack_lb_loadbalancer_v2.tf_lb.id #restrict the access of the listener to the private network subnet allowed_cidrs = [openstack_networking_subnet_v2.tf_lb_subnet.cidr] } ``` #### Configure Prometheus to collect your metrics Once the `PROMETHEUS` listener is `ACTIVE`, you can configure Prometheus to collect metrics from the Load Balancer by updating the `prometheus.yml` file. ```yaml [scrape_configs] - job_name: 'Octavia LB1' static_configs: - targets: ['192.0.2.10:9100'] ``` For more information on setting up Prometheus, see the [Prometheus project website](https://prometheus.io/). You can connect [Grafana](https://grafana.com) to the [Prometheus](https://prometheus.io) instance to provide additional graphing and dashboard capabilities. A Grafana dashboard for Octavia Load Balancers is available [here](https://grafana.com/grafana/dashboards/15828-openstack-octavia-amphora-load-balancer/). The metrics have the following naming format: - `octavia_loadbalancer_xxx` or `octavia_memory_pool_xxx` metrics are global to the load balancer - `octavia_listener_xxx` metrics are instantiated per listener using the label \{listener=ID} - `octavia_pool_xxx` metrics are instantiated per pool using the label \{pool=ID} - `octavia_member_xxx` metrics are instantiated per member using the label \{member=ID} | Metric | Description | | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | octavia\_loadbalancer\_cpu | Load balancer CPU utilization (percentage). | | octavia\_loadbalancer\_memory | Load balancer memory utilization (percentage). | | octavia\_memory\_pool\_allocated\_bytes | Total amount of memory allocated in the memory pools (in bytes). | | octavia\_memory\_pool\_used\_bytes | Total amount of memory used in the memory pools (in bytes). | | octavia\_memory\_pool\_failures\_total | Total number of failed memory pool allocations. | | octavia\_loadbalancer\_max\_connections | Maximum number of concurrent connections. | | octavia\_loadbalancer\_current\_connections | Number of active sessions. | | octavia\_loadbalancer\_connections\_total | Total number of created sessions. | | octavia\_loadbalancer\_requests\_total | Total number of requests (TCP or HTTP). | | octavia\_loadbalancer\_max\_ssl\_connections | Configured maximum number of concurrent SSL connections. | | octavia\_loadbalancer\_current\_ssl\_connections | Number of active SSL connections. | | octavia\_loadbalancer\_ssl\_connections\_total | Total number of opened SSL connections. | | octavia\_loadbalancer\_current\_connection\_rate | Current number of connections per second over last elapsed second. | | octavia\_loadbalancer\_limit\_connection\_rate | Configured maximum number of connections per second. | | octavia\_loadbalancer\_max\_connection\_rate | Maximum observed number of connections per second. | | octavia\_loadbalancer\_current\_session\_rate | Current number of sessions per second over last elapsed second. | | octavia\_loadbalancer\_limit\_session\_rate | Configured maximum number of sessions per second. | | octavia\_loadbalancer\_max\_session\_rate | Maximum observed number of sessions per second. | | octavia\_loadbalancer\_current\_ssl\_rate | Current number of SSL sessions per second over last elapsed second. | | octavia\_loadbalancer\_limit\_ssl\_rate | Configured maximum number of SSL sessions per second. | | octavia\_loadbalancer\_max\_ssl\_rate | Maximum observed number of SSL sessions per second. | | octavia\_loadbalancer\_current\_frontend\_ssl\_key\_rate | Current frontend SSL Key computation per second over last elapsed second. | | octavia\_loadbalancer\_max\_frontend\_ssl\_key\_rate | Maximum observed frontend SSL Key computation per second. | | octavia\_loadbalancer\_current\_backend\_ssl\_key\_rate | Current backend SSL Key computation per second over last elapsed second. | | octavia\_loadbalancer\_max\_backend\_ssl\_key\_rate | Maximum observed backend SSL Key computation per second. | | octavia\_loadbalancer\_ssl\_cache\_lookups\_total | Total number of SSL session cache lookups. | | octavia\_loadbalancer\_ssl\_cache\_misses\_total | Total number of SSL session cache misses. | | octavia\_loadbalancer\_http\_comp\_bytes\_in\_total | Number of bytes per second over last elapsed second, before http compression. | | octavia\_loadbalancer\_http\_comp\_bytes\_out\_total | Number of bytes per second over last elapsed second, after http compression. | | octavia\_loadbalancer\_limit\_http\_comp | Configured maximum input compression rate in bytes. | | octavia\_loadbalancer\_listeners | Current number of active listeners. | | octavia\_loadbalancer\_dropped\_logs\_total | Total number of dropped logs. | | octavia\_listener\_status | Current status of the listener. 0=OFFLINE, 1=ONLINE, 2=DEGRADED. | | octavia\_listener\_current\_sessions | Current number of active sessions. | | octavia\_listener\_max\_sessions | Maximum observed number of active sessions. | | octavia\_listener\_limit\_sessions | Configured session limit. | | octavia\_listener\_sessions\_total | Total number of sessions. | | octavia\_listener\_limit\_session\_rate | Configured limit on new sessions per second. | | octavia\_listener\_max\_session\_rate | Maximum observed number of sessions per second. | | octavia\_listener\_connections\_rate\_max | Maximum observed number of connections per second. | | octavia\_listener\_connections\_total | Total number of connections. | | octavia\_listener\_bytes\_in\_total | Current total of incoming bytes. | | octavia\_listener\_bytes\_out\_total | Current total of outgoing bytes. | | octavia\_listener\_requests\_denied\_total | Total number of denied requests. | | octavia\_listener\_responses\_denied\_total | Total number of denied responses. | | octavia\_listener\_request\_errors\_total | Total number of request errors. | | octavia\_listener\_denied\_connections\_total | Total number of requests denied by connection rules. | | octavia\_listener\_denied\_sessions\_total | Total number of requests denied by session rules. | | octavia\_listener\_failed\_header\_rewriting\_total | Total number of failed header rewriting rules. | | octavia\_listener\_http\_requests\_rate\_max | Maximum observed number of HTTP requests per second. | | octavia\_listener\_http\_requests\_total | Total number of HTTP requests received. | | octavia\_listener\_http\_responses\_total | Total number of HTTP responses. | | octavia\_listener\_intercepted\_requests\_total | Total number of intercepted HTTP requests. | | octavia\_listener\_http\_cache\_lookups\_total | Total number of HTTP cache lookups. | | octavia\_listener\_http\_cache\_hits\_total | Total number of HTTP cache hits. | | octavia\_listener\_http\_comp\_bytes\_in\_total | Total number of HTTP response bytes fed to the compressor. | | octavia\_listener\_http\_comp\_bytes\_out\_total | Total number of HTTP response bytes emitted by the compressor. | | octavia\_listener\_http\_comp\_bytes\_bypassed\_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). | | octavia\_listener\_http\_comp\_responses\_total | Total number of HTTP responses that were compressed. | | octavia\_pool\_status | Current status of the pool. 0=OFFLINE, 1=ONLINE. | | octavia\_pool\_current\_sessions | Current number of active sessions. | | octavia\_pool\_max\_sessions | Maximum observed number of active sessions. | | octavia\_pool\_limit\_sessions | Configured session limit. | | octavia\_pool\_sessions\_total | Total number of sessions. | | octavia\_pool\_max\_session\_rate | Maximum observed number of sessions per second. | | octavia\_pool\_last\_session\_seconds | Number of seconds since last session assigned to a member. | | octavia\_pool\_current\_queue | Current number of queued requests. | | octavia\_pool\_max\_queue | Maximum observed number of queued requests. | | octavia\_pool\_connection\_attempts\_total | Total number of connection establishment attempts. | | octavia\_pool\_connection\_reuses\_total | Total number of connection reuses. | | octavia\_pool\_bytes\_in\_total | Current total of incoming bytes. | | octavia\_pool\_bytes\_out\_total | Current total of outgoing bytes. | | octavia\_pool\_queue\_time\_average\_seconds | Avg. queue time for last 1024 successful connections. | | octavia\_pool\_connect\_time\_average\_seconds | Avg. connect time for last 1024 successful connections. | | octavia\_pool\_response\_time\_average\_seconds | Avg. response time for last 1024 successful connections. | | octavia\_pool\_total\_time\_average\_seconds | Avg. total time for last 1024 successful connections. | | octavia\_pool\_max\_queue\_time\_seconds | Maximum observed time spent in the queue | | octavia\_pool\_max\_connect\_time\_seconds | Maximum observed time spent waiting for a connection to complete | | octavia\_pool\_max\_response\_time\_seconds | Maximum observed time spent waiting for a member response. | | octavia\_pool\_max\_total\_time\_seconds | Maximum observed total request+response time (request+queue+connect+response+processing) | | octavia\_pool\_requests\_denied\_total | Total number of denied requests. | | octavia\_pool\_responses\_denied\_total | Total number of denied responses. | | octavia\_pool\_connection\_errors\_total | Total number of connection errors. | | octavia\_pool\_response\_errors\_total | Total number of response errors. | | octavia\_pool\_retry\_warnings\_total | Total number of retry warnings. | | octavia\_pool\_redispatch\_warnings\_total | Total number of redispatch warnings. | | octavia\_pool\_failed\_header\_rewriting\_total | Total number of failed header rewriting warnings. | | octavia\_pool\_client\_aborts\_total | Total number of data transfers aborted by the client. | | octavia\_pool\_member\_aborts\_total | Total number of data transfers aborted by the server. | | octavia\_pool\_active\_members | Current number of active members. | | octavia\_pool\_backup\_members | Current number of backup members. | | octavia\_pool\_check\_up\_down\_total | Total number of UP->DOWN transitions. | | octavia\_pool\_check\_last\_change\_seconds | Number of seconds since the last UP\<->DOWN transition. | | octavia\_pool\_downtime\_seconds\_total | Total downtime (in seconds) for the pool. | | octavia\_pool\_loadbalanced\_total | Total number of times a pool was selected, either for new sessions, or when redispatching. | | octavia\_pool\_http\_requests\_total | Total number of HTTP requests received. | | octavia\_pool\_http\_responses\_total | Total number of HTTP responses. | | octavia\_pool\_http\_cache\_lookups\_total | Total number of HTTP cache lookups. | | octavia\_pool\_http\_cache\_hits\_total | Total number of HTTP cache hits. | | octavia\_pool\_http\_comp\_bytes\_in\_total | Total number of HTTP response bytes fed to the compressor. | | octavia\_pool\_http\_comp\_bytes\_out\_total | Total number of HTTP response bytes emitted by the compressor. | | octavia\_pool\_http\_comp\_bytes\_bypassed\_total | Total number of bytes that bypassed the HTTP compressor (CPU/BW limit). | | octavia\_pool\_http\_comp\_responses\_total | Total number of HTTP responses that were compressed. | | octavia\_member\_status | Current status of the member. 0=ERROR, 1=ONLINE, 2=OFFLINE, 3=DRAIN. | | octavia\_member\_current\_sessions | Current number of active sessions. | | octavia\_member\_max\_sessions | Maximum observed number of active sessions. | | octavia\_member\_limit\_sessions | Configured session limit. | | octavia\_member\_sessions\_total | Total number of sessions. | | octavia\_member\_max\_session\_rate | Maximum observed number of sessions per second. | | octavia\_member\_last\_session\_seconds | Number of seconds since last session assigned to the member. | | octavia\_member\_current\_queue | Current number of queued requests. | | octavia\_member\_max\_queue | Maximum observed number of queued requests. | | octavia\_member\_queue\_limit | Configured maxqueue for the member (0 meaning no limit). | | octavia\_member\_bytes\_in\_total | Current total of incoming bytes. | | octavia\_member\_bytes\_out\_total | Current total of outgoing bytes. | | octavia\_member\_queue\_time\_average\_seconds | Avg. queue time for last 1024 successful connections. | | octavia\_member\_connect\_time\_average\_seconds | Avg. connect time for last 1024 successful connections. | | octavia\_member\_response\_time\_average\_seconds | Avg. response time for last 1024 successful connections. | | octavia\_member\_total\_time\_average\_seconds | Avg. total time for last 1024 successful connections. | | octavia\_member\_max\_queue\_time\_seconds | Maximum observed time spent in the queue | | octavia\_member\_max\_connect\_time\_seconds | Maximum observed time spent waiting for a connection to complete | | octavia\_member\_max\_response\_time\_seconds | Maximum observed time spent waiting for a member response. | | octavia\_member\_max\_total\_time\_seconds | Maximum observed total request+response time (request+queue+connect+response+processing) | | octavia\_member\_connection\_attempts\_total | Total number of connection establishment attempts. | | octavia\_member\_connection\_reuses\_total | Total number of connection reuses. | | octavia\_member\_responses\_denied\_total | Total number of denied responses. | | octavia\_member\_connection\_errors\_total | Total number of connection errors. | | octavia\_member\_response\_errors\_total | Total number of response errors. | | octavia\_member\_retry\_warnings\_total | Total number of retry warnings. | | octavia\_member\_redispatch\_warnings\_total | Total number of redispatch warnings. | | octavia\_member\_failed\_header\_rewriting\_total | Total number of failed header rewriting warnings. | | octavia\_member\_client\_aborts\_total | Total number of data transfers aborted by the client. | | octavia\_member\_server\_aborts\_total | Total number of data transfers aborted by the server. | | octavia\_member\_weight | Member weight. | | octavia\_member\_check\_failures\_total | Total number of failed check (Only counts checks failed when the member is up). | | octavia\_member\_check\_up\_down\_total | Total number of UP->DOWN transitions. | | octavia\_member\_downtime\_seconds\_total | Total downtime (in seconds) for the member. | | octavia\_member\_check\_last\_change\_seconds | Number of seconds since the last UP\<->DOWN transition. | | octavia\_member\_current\_throttle | Current throttle percentage for the member, when slowstart is active, or no value if not in slowstart. | | octavia\_member\_loadbalanced\_total | Total number of times a member was selected, either for new sessions, or when redispatching. | | octavia\_member\_http\_responses\_total | Total number of HTTP responses. | | octavia\_member\_idle\_connections\_current | Current number of idle connections available for reuse | | octavia\_member\_idle\_connections\_limit | Limit on the number of available idle connections | ## Go further If you are interested in automating this scenario with terraform and with our managed Grafana service, you can find ready to run example on this [github repo](https://github.com/yomovh/tf-at-ovhcloud) Visit our dedicated Discord channel: [https://discord.gg/ovhcloud](https://discord.gg/ovhcloud). Ask questions, provide feedback and interact directly with the team that builds our databases services. If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project. Join our [community of users](https://community.ovhcloud.com/).