Monitoring and Alerting Archives - SearchStax Docs

Performance Monitoring Overview

Tom Humbarger — Wed, 10 May 2023 16:36:09 +0000

Monitoring and Alerting Overview

Overview

The SearchStax Managed Search service helps you maintain the health and reliability of your Solr deployment with active monitoring and alerting.

Questions?

Do not hesitate to contact the SearchStax Support Desk.

The post Performance Monitoring Overview appeared first on SearchStax Docs.

Performance Monitoring

admin — Wed, 03 Mar 2021 10:03:26 +0000

Performance Monitoring

Overview

The SearchStax Managed Search service provides real-time and historical statistical graphs of critical Solr server and JVM metrics and statistics. These statistics provide a birds-eye view of Solr cluster health, enabling you to tune server performance.

Monitoring Data Retention by Support Level Agreement (SLA)

“Silver” (self-service) accounts retain monitoring data for 24 hours.
“Gold” accounts retain monitoring data for 7 days.
“Platinum” accounts retain monitor data for 30 days.
“Platinum Plus” accounts retain monitoring data for 30 days or more, depending on the contract.

Please see our Pricing page for the most up-to-date information on feature availability for each SLA tier.

Don’t use periods in collection names!

The Pulse monitoring feature cannot display the history of a collection containing a period in the collection name.

Contents of this page:

How to Access the Monitoring Graphs
About the Graphs
Availability Report
Alerting

How to Access the Monitoring Graphs

You can access the graphs from the Deployment Details screen.

To access the graphs:

From the SearchStax Managed Search dashboard, select Dedicated Infrastructure.
Choose a deployment from the deployment list.
Select Monitoring, and then CPU, Memory, JVM.
Check the nodes to view from the list of nodes. (Note that the Solr nodes of a NextGen cluster are nodes 4 and 5.)

Set a time interval from the interval button in the upper right. Note that short intervals display more quickly than long intervals.

Note the date and time controls at the top of the dialog box. These let you examine data from a small time interval that occurred days or weeks in the past.

About the Graphs

Managed Search provides graphs for the following server and search metrics.

CPU, Memory, JVM

These are monitoring graphs from a server that is running out of memory. Pay close attention to the Y-axis labels. The graph zooms in on the data, so the Y-axis is dynamic.

The following server-performance metrics are displayed:

Metric	Definition
Usage	CPU load as a percent of maximum. (See Is 100% CPU a bad thing?)
System Load Average	Number of processes per CPU over one-minute intervals. Values below 1.0 indicate that processes are not waiting.
Memory	Total memory size and used memory size plotted on a single graph. (See How do I recognize a Solr Out-of-Memory Error?)
Swap Usage	Total disc swap space available and used over time.
JVM Heap Memory Usage	Heap memory usage over time. Heap memory is used for storing Java objects. When heap memory reaches 100%, Solr stops running.
Non-Heap Memory	Non-heap memory usage over time. Non-heap memory is used by Java to store loaded classes and other metadata.
JVM Thread Count	Server thread count usage over time.
1 min 5XX Error Rate	5XX errors summed over one-minute intervals.
Disk Usage	Disk space usage in gigabytes. The dashed line is the maximum available.

Collections

The Collections table shows the health of replicas across servers.

Garbage Collection

The following garbage-collection metrics are reported by Managed Search:

Metric	Definition
ConcurrentMarkSweep CollectionTime & CollectionCount	Collection time and count for the ConcurrentMarkSweep approach.
ParNew CollectionTime & CollectionCount	Collection time and count for the ParNew approach.

Search Throughput and Latency

Managed Search provides graphs that report on throughput and latency.

Metric	Definition
Average Requests/second (ms)	Reports average search requests per second.
Average Time/Request (ms)	Reports average time for search request to fulfill the query.
5-minute Request Rate	Reports search-request rate over a 5-minute interval.
15-minute Request Rate	Reports search-request rate over 15-minute interval.
Timeouts	Reports server timeouts, where the server did not respond to the request fast enough.
Errors	Numbers of search-request errors.

Indexing Throughput and Latency

Managed Search provides graphs that report on throughput and latency.

Metric	Definition
Average Requests/Second	Reports average indexing requests per second.
Average Time/Request (ms)	Reports average time for indexing request to fulfill the query.
Timeouts	Reports server timeouts, where the server did not respond to the request fast enough.
Errors	Numbers of indexing request errors.
Update Handler graphs	Reports various Update Handler metrics over a period of time, such as commits, autoCommits, deletesById, etc.

Caches

SearchStax Managed Search provides graphs that report on Caches.

Metric	Definition
QueryResultCache	This cache stores ordered sets of document IDs — the top N results of a query ordered by some criterion. Multiple metrics report how the query result cache is being utilized to fulfill search requests. Metrics include hit ratio, evictions, warmup time, etc.
FilterCache	This cache stores unordered sets of document IDs that match the queries. Multiple metrics report how the filter cache is being utilized to fulfill search requests. Metrics include itratio, evictions, warmup time, etc.
DocumentCache	The documentCache stores Lucene Document objects that have been fetched from disk. Multiple metrics report how the document cache is being utilized. Metrics include itratio, evictions, warmup time, etc.
FieldValueCache	The fieldValueCache is similar to fieldCache, and supports multiple values per document. It is primarily used for faceting. Metrics include warmup time, size.

Service – Total Requests

Clients running clustered deployments may view one more graph. The Service graph displays the number of requests received by the cluster’s Load Balancer over time. This can reveal cyclic patterns of high demand:

“Requests” include all external messages directed to the Solr cluster, such as /select and /update messages plus API calls and Pulse monitoring.

Availability Report

Premium clients (Platinum, and Platinum Plus SLAs) get access to an Availability Report in the SearchStax dashboard. This is a graph and a summary that documents response time and downtime so they can be compared with the guarantees of the SLA.

Note that this feature is accessed through the Availability Report screen of the SearchStax Managed Search dashboard.

Alerting

The SearchStax Managed Search alerting features are described on the Solr Service Alerting page.

Questions?

Do not hesitate to contact the SearchStax Support Desk.

The post Performance Monitoring appeared first on SearchStax Docs.

Email Alerting

admin — Wed, 03 Mar 2021 10:50:26 +0000

Email Alerts

Overview

The SearchStax Managed Search service provides two kinds of real-time email alerts:

Heartbeat alerts: Notify a list of email recipients when a server starts or stops operating.
Threshold alerts: Notify a list of email recipients when a server exceeds a performance threshold.

Either type of alert may optionally invoke a webhook to notify an external bug-tracking system or alerting system.

Both types of alerts create an “incident” report that you can inspect in the SearchStax Managed Search dashboard.

Alerts send a follow-up email when the condition is resolved.

Heartbeat Alerts on All Servers

SearchStax automatically adds Heartbeat Alerts to all new Solr Cloud servers — both Solr and Zookeeper.

The alerts send email to all users any time the server is silent for five full minutes. A second email is sent when the server becomes active again.

Four Threshold Alerts on Solr Servers

SearchStax automatically configures four standard alerts on all new Solr nodes. Account users are notified by email when:

Search Timeouts exceed 10 in 5 minutes
JVM Heap Memory exceeds 80% for 5 minutes
Index Error Count exceeds 10 in 5 minutes
Disk Space Used exceeds 80% for 5 minutes

A second email is sent when the metric falls below the threshold for five minutes.

Premium Alerting

For SearchStax customers with Premium Support Level Agreements (SLAs), we have an internal monitoring system that notifies our on-call support team of any issues.

Contents of this page:

Popular Alerts

SearchStax customers often implement some or all of the following alerts on a production system:

Alert	Node	Trigger	Delay	Max Alerts	Repeat
Heartbeat	All	–	5 min	1	15 min
CPU Usage	Solr	>80%	5 min	1	15 min
Free Disk Space	Solr	<20%	5 min	1	15 min
JVM Heap Used	Solr	>80%	5 min	1	15 min
Index Error Count *	Solr	>10	5 min	1	15 min
Index Timeout	Solr	>10	5 min	1	15 min
Index Average Response Time / Request	Solr	>60000	5 min	1	15 min
Search Timeouts	Solr	>10	5 min	1	15 min
Search Average Response Time / Request	Solr	>3000ms	5 min	1	15 min

* Note that index-error alerts often mean that some of your documents have been dropped from the index. See What Causes Indexing Errors?

Alerts for Query Errors

Clients who are concerned about Search Errors sometimes add these two threshold alerts in addition to the standard alerts in the previous section:

Alert	Node	Trigger	Delay	Max Alerts	Repeat
1 Min. 5XX Error Rate	Solr	> 1	5 min	1	15 min
Search Error Count	Solr	> 10	5 min	1	15 min

The 1 Min. 5XX Error Rate may alert you to query syntax errors (among other issues). These queries are rejected by Solr. Check your Solr log files. The Search Error Count responds to syntactically-correct queries that have schema errors such as unknown fields. As with all threshold alerts, you will have to experiment to find settings you can live with.

Heartbeat Alerts

Both Zookeeper and Solr send reports of system metrics to SearchStax once per minute. You can set up a “heartbeat” alert to notify you if these reports are interrupted. The system also notifies you when the updates resume.

Set up a Heartbeat Alert

To set up a heartbeat alert, open the SearchStax Managed Search dashboard.

Click the Dedicated Infrastructure label in the left-side navigation menu.
Select a Deployment.
Open the Alerting menu.
Select Heartbeat.

To create a new Heartbeat Alert, click the New Heartbeat button.

Control	Description
Server	The Server control offers a list of the servers in this deployment. Select one of them to monitor.
Name	Give the alert a name that you will recognize when you see it in email.
Notify if data is missing for more than…	When heartbeat data stops flowing, wait this long before triggering the alert.
Max Notifications	Alert emails are reissued every two minutes. How many of them do you want to send?
Send alerts to	Choose from a list of registered SearchStax users.
Send trigger alert to webhook	Invoke this webhook when this alert is triggered.
Send resolve alert to webhook	Invoke this webhook when the alert is resolved.

Heartbeat Email

A heartbeat email notification resembles this one:

Dear SearchStax Customer,

The alert ss123456-5 heartbeat alert for your deployment Films (ss123456) has been triggered.

The following host is unreachable.

Host: ss123456-5

To View Metrics in Dashboard: https://app.searchstax.com/admin/deployment/pulse/deployment/ss123456/alert/incident/update/65737

To Edit this Alert: https://app.searchstax.com/admin/deployment/pulse/deployment/ss123456/alert/heartbeat/update/841/

This alert was triggered at 2020-01-15 20:12:27 UTC.

This alert was raised for account AccountName.

You will receive a similar “UP” notification when the heartbeat is again detected.

Threshold Alerts

A “threshold” alert watches a specific system metric and sends you email when the metric meets or exceeds some value.

Managed Search allows you to monitor the following metrics:

Total Requests – can be applied to the App Gateway server only.
CPU Usage
JVM Thread Count
Disk Space Used
Disk Space Free
JVM Heap Memory Used
1 Min. 5XX Error Rate
Swap Used
System Load Average
Search – Avg. Requests/s
Search – 5 Min. Request Rate
Search Timeouts
Search Error Count
Index – Timeouts
Index – Error Count
QueryResultCache – evictions
QueryResultCache – warmupTime
QueryResultCache – hit ratio
Filtercache – evictions
Filtercache – warumpTime
Filtercache – hit ratio
DocumentCache – evictions
DocumentCache – hit ratio
DocumentCache – warmupTime
FieldValueCache – evictions
FieldValueCache – hit ratio
FieldValueCache – warmupTime
Search – Avg. Response Time/Request (ms)
Index – Avg Response Time/Request (ms)
JVM Non-Heap Memory Used
Physical Memory Used
Index – 5 min. Request Rate

Set up a Threshold Alert

To set up a threshold alert, open the SearchStax Managed Search dashboard and navigate to a specific deployment.

Click the Dedicated Infrastructure label in the left-side navigation menu.
Select a Deployment.
Open the Alerting menu.
Select Threshold.

To create a new Threshold Alert, click the Create New Alert button.

Control	Description
Host Machine	The Host Machine control offers a list of the servers in this deployment. Select one of them to monitor.
Metric Name	Choose one of many internal metrics monitored by Managed Search.
Collection	Some metrics are collection-specific. Others apply to “all collections.”
Alert Name	Give the alert a name that you will recognize when you see it in email.
Delay of at least	Metric must exceed threshold for this long before triggering the alert.
Max Alerts	Alert emails are reissued every two minutes. How many of them do you want to send?
Repeat Every	Time to wait between sending repeat email messages.
Send alerts to	Choose from a list of registered SearchStax users.
Send trigger alert to webhook	Invoke this webhook when this alert is triggered.
Send resolve alert to webhook	Invoke this webhook when the alert is resolved.

Receive a Threshold Alert

A threshold email notification resembles this one:

Dear SearchStax Customer,

The alert "Server 5 below 10% CPU" for your deployment Films (ss123456) has been triggered.

Host:           ss123456-5
Metric:         CPU Usage
Name:           "Server 5 below 10% CPU"
Threshold:      < 10.0%
Current Value:  0.01 %

To View Metrics in Dashboard: https://app.searchstax.com/admin/deployment/pulse/deployment/ss123456/system/

To Edit this Alert: https://app.searchstax.com/admin/deployment/pulse/deployment/ss123456/alert/incident/update/6012

This alert was triggered at 2019-12-20 17:51:42 UTC.

This alert was raised for account AccountName.

Incidents

To view a list of your heartbeat or threshold incidents:

Click the Dedicated Infrastructure label in the left-side navigation bar.
Select a Deployment from the list.
Open the Alerting menu.
Select Incidents.

Click the incident to view its details. You’ll see a brief description of the incident followed by a timeline of events. Read the timeline from the bottom up.

Opting Out

Not everyone wants to receive email alerts. If you have users who find the alerts annoying, they can opt out.

Each alert has its own list of recipients. If someone complains about a specific alert, you can remove them from that alert’s notification list. Alternately, you could raise the alert threshold, or increase the alert’s activation time. Either action should reduce the number of complaints about email alerts.

Questions?

Do not hesitate to contact the SearchStax Support Desk.

The post Email Alerting appeared first on SearchStax Docs.

Webhooks

admin — Thu, 04 Mar 2021 04:58:55 +0000

Webhooks

Overview

The SearchStax Managed Search service can POST Heartbeat and Threshold Alerts to the webhook endpoints of external Alerting and Incident Management systems. See our blog SearchStax Webhooks: Better Solr Alerting through Extensibility for background on webhook features.

The webhook messages are invoked by Heartbeat and Threshold Alerts, and can carry a payload of data about the alert and the system that triggered it.

Contents of this page:

What is a Webhook?
Defining a Webhook
Adding Webhooks to an Alert
Predefined Variables

What is a Webhook?

Webhooks are user-defined HTTP callbacks. They are usually triggered by some event, such as an alert changing status in SearchStax. When the event occurs, the source site (SearchStax) makes an HTTP request to the URL configured for the webhook. Events on one site can invoke behavior on another.

A common use is to notify bug-tracking systems and alerting systems, although the SearchStax webhook mechanism can reach out to any web application that has a webhook endpoint.

Defining a Webhook

A webhook must be defined in the SearchStax Managed Search Dashboard before it can be referenced in an alert. Look for the Webhooks entry in the navigation pane on the left margin of the page.

Click the Create Webhook button to open the webhook editor.

Control	Description
Name	The name of the webhook. It will appear in a droplist of webhooks in the alerts editor.
URL	The URL of the webhook endpoint of the target application.
Ignore SSL Validation	Sometimes the target of your webhook post is using SSL with a self-signed certificate. This setting overrides the certificate check.
Paused	Deactivate this webhook without having to delete it.
Payload	The payload is a JSON list of field keys and values. The fields are the labels expected by the target application. The values are predefined SearchStax variables.
Update	The Update button saves the webhook and closes the editor.

Adding Webhooks to an Alert

The alert editor offers droplists that let you associate your webhooks with specific alerts.

Note that you can send one webhook when the alert is triggered, and then send a different webhook when the alert is closed.

Predefined Variables

SearchStax email alerts report various internal values to the user. These values are available as predefined variables to put into the payload of the webhook.

Variable	Description	Example
$ALERT_CURRENT_VALUE	The current value of the metric.	“0.01”
$ALERT_ID	The internal ID number of the alert.	“6012”
$ALERT_METRIC	The name of the alert metric.	“os.SystemCpuLoad”
$ALERT_STATUS	Open or closed.	“OPEN”
$ALERT_THRESHOLD_VALUE	The threshold.	“10.0”
$ALERT_THRESHOLD_OPERATOR	The relation between the current value and the threshold.	“>”
$ALERT_TITLE	The name of the alert.	“Server 5 below 10% CPU”
$ALERT_TYPE	Heartbeat or Threshold.	“Threshold”
$DATE	The time in UTC.	“2019-12-20T18:18:39+00:00”
$DEPLOYMENT_NAME	The name of the deployment.	“Films”
$DEPLOYMENT_UID	The ID number of the SearchStax deployment.	“ss123456”
$HOSTNAME	The name of the server that issued the alert.	“ss123456-5”

Contact Us!

Don’t hesitate to reach out to the SearchStax Support Desk. We are happy to answer your questions.

The post Webhooks appeared first on SearchStax Docs.