This notice is not as alarming as it looks. Like a Heartbeat Alert, it means that the Pulse server at SearchStax has not heard from the Pulse Agent for several minutes. It does not necessarily mean that the server is down or that Solr service has been interrupted.

For one thing, when you restart Solr, the “Pulse Agent is Down” warning may be visible for a few minutes. Give it some time to come on line after the interruption.

When you see this warning, always check the CPU Monitoring graphs for the server to verify that the Solr service is still up. An interruption in service will appear as a gap in the graph.

If the warning persists, contact the SearchStax Support desk to investigate the reason.

Questions?

Do not hesitate to contact the SearchStax Support Desk.

The post Pulse Agent is Down! appeared first on SearchStax Docs.

These images were taken from a SearchStax NDN1 single-node deployment before any client configuration or data ingestion.

An NDN1 deployment is a small node for configuration testing. Clients use these nodes for proof-of-concept demos. The great virtue of the NDN1 is that it is inexpensive and disposable. You can run it for less than $1.50/day on the pay-as-you-go plan, and you can terminate it at will. An NDN1 isn’t meant to handle large data sets, a high commit rate, or a high query load.

In a NDN1, the solr JVM occupies half of the server’s total memory. Solr may reserve additional memory for utility tasks related to index building. The operating system and other processes, including Zookeeper, share the remaining memory.

If the physical memory on the box hits 100%, the swap space on the server will start to be used. The server will not run out of memory unless the physical memory and swap memory both approach 100%.

However, if the Solr JVM usage reaches 100%, the Solr process will stop. For Azure or Google Cloud deployments, SearchStax will restart Solr automatically. For a deployment on AWS, you can use the Stop Solr/Start Solr links on the deployment details page to restart Solr. If you have any difficulty, contact the SearchStax Support Desk. We’ll restart Solr for you and help you understand your options.

See the following pages for more information:

• How do I fix an Out of Memory Error?
• Solr Performance Monitoring.

The post Physical memory usage is over 90%! appeared first on SearchStax Docs.

• Pulse Monitoring: All SearchStax Managed Search deployments track Solr performance statistics such as:
  • CPU, Memory, JVM
  • Collections
  • Garbage Collection
  • Search Throughput and Latency
  • Indexing Throughput and Latency
  • Caches
• Email Alerting: All Dedicated Deployments provide a mechanism for heartbeat and threshold-based email alerts triggered by a long list of metrics.
• SearchStax Site Search solution: The SearchStax Site Search product tracks end-user search events. It lets you make real-time adjustments to search behavior and presentation.
• Availability Reporting: Premium clients have access to a graph depicting system availability in relation of the uptime guarantees of the service-level agreement.
• Internal Monitoring Support: SearchStax support engineers receive automatic alerts about memory and performance issues in deployments covered by premium service-level agreements (Gold, Platinum, and Platinum Plus levels).

The post Does SearchStax monitor my Solr deployments? appeared first on SearchStax Docs.

CPU Usage alerts can result when the system is under heavy stress.

See Is 100% CPU a Bad Thing? for a discussion of how an overloaded CPU can cause cascading failures that can bring down a cluster.

Overloading is often associated with inappropriate indexing configurations. Consider adjusting your commit strategy to place less load on the system.

This situation can be relieved (not necessarily cured) by performing a rolling restart to clear the queues. Contact SearchStax Support if you need any help there.

The post CPU Usage appeared first on SearchStax Docs.

System Load Average alerts can result when the system is under heavy stress. Check the System Load graphs to evaluate the load.

Most Solr deployments can handle a load of 1 or 2. Some can handle more, but no system can handle a load of 800 as shown in the graph above.

Overloading is often associated with inappropriate indexing configurations. Consider adjusting your commit strategy to place less load on the system.

This situation can be relieved (not necessarily cured) by performing a rolling restart to clear the queues. Contact SearchStax Support if you need any help there.

If excessive system load becomes a frequent problem, the deployment might need an upgrade.

The post System Load Average appeared first on SearchStax Docs.

This condition arises when the system is overloaded, or when the replication mechanism has broken down.

The graph above shows a very sluggish cluster and the impact of a rolling restart.

System Load is Too High

Index Average Response Time / Request alerts can result when the system is under heavy stress. Check the System Load/CPU graphs to evaluate the load. Also, check the Service Graph to see if there is a spike.

This is often associated with a heavy episode of indexing. Solr Admin > Cloud > Tree > Overseer can show if there are too many items queued up. If the alerts are associated with indexing events, consider throttling back the flow of /update messages.

This situation can be relieved by performing a rolling restart to clear the queues. Contact SearchStax Support if you need any help there.

If the issue is because of System load and is consistent, the deployment might need an upgrade.

Replica In Unhealthy State

It is possible for a replica to lose synchronization with the rest of the collection, putting it into “recovery” mode while Zookeeper attempts to rebuild it. A replica in recovery is not available for index updates, which may cause Index Average Response Time / Request errors.

Check the Solr Admin UI > Cloud > Graph to see if all collections are replicated on all solr nodes and are in a healthy state. If there is a problem, contact SearchStax Support for assistance.

The post Index Average Response Time / Request appeared first on SearchStax Docs.

An Index Error aborts the indexing of a document. That document is lost to the system until resubmitted.

Most Index Errors are due to a mismatch between the collection’s schema file and the fields (sometimes values) of the incoming documents.

Missing Fields in Schema

Index Errors are often caused by an incomplete or out-of-date schema. The schema is the specification for the Solr index. Often the client adds fields to the CMS (such as Sitecore) without updating the Solr schema. The CMS subsequently sends updates to Solr that include fields that Solr doesn’t know. Examine your solr.log files for entries that mention “unknown field,” “undefined field,” “required field,” “mandatory field,” or “adding field,” among others.

The fix is to update the schema by adding the missing fields, or by repopulating the schema(s) from Sitecore. Note that you must reload your data after changing the schema.

Replica In Unhealthy State

It is possible for a replica to lose synchronization with the rest of the collection, putting it into “recovery” mode while Zookeeper attempts to rebuild it. A replica in recovery is not available for index updates, which may cause Index Errors.

Check the Solr Admin UI > Cloud > Graph to see if all collections are replicated on all solr nodes and are in a healthy state. If there is a problem, contact SearchStax Support for assistance.

The post Index Error Count appeared first on SearchStax Docs.

An indexing request times out after 180 seconds. You get an error message in the solr.log file, but the incoming documents are not lost. They are queued up in Zookeeper.

This usually means that the Solr system is overloaded and is just too busy to process index requests in a timely manner.

System Load is Too High

Index Timeout alerts can result when the system is under heavy stress. Check the System Load/CPU graphs to evaluate the load. Also, check the Service Graph to see if there is a spike.

This is often associated with a heavy episode of indexing. Solr Admin > Cloud > Tree > Overseer can show if there are too many items queued up. If the alerts are associated with indexing events, consider throttling back the flow of /update messages.

This situation can be relieved by performing a rolling restart to clear the queues. Contact SearchStax Support if you need any help there.

Query Volume is Too High

It is possible for a cluster to become too busy handling search events, such as during a holiday spike in traffic. Evaluate if the number of timeouts is decreasing and one should just wait, else open a ticket with SearchStax support.

The post Index Timeout appeared first on SearchStax Docs.

If the JVM level reaches 100%, Solr will stop, leaving behind an OOM-killer log file. Azure and GCP deployments will usually restart automatically. AWS deployments may need to be manually stopped and restarted.

Heavy Indexing Load

High JVM is often associated with a heavy episode of indexing, either in terms of the number of /update requests, number of documents per request, or the size of the documents.

Excessive commit requests can slow indexing down to the point the JVM fills up.

If the alerts are associated with indexing events, consider throttling back the flow of /update messages.

This situation can be relieved by performing a rolling restart (as shown above). Contact SearchStax Support if you need any help there.

Heavy Query Load

Queries that involve faceting, grouping, or sorting of results can fill up JVM. So can queries that involve deep pagination of results.

One common problem is queries that request too many results. Sitecore, for instance, asks for one million result items on every query as a default. Solr allocates memory for every item, needlessly filling up JVM.

Deployment Upgrade

Solr projects that constantly index new data will eventually outgrow their Managed Search deployments. Chronic JVM issues might mean that it is time to upgrade. Please open a ticket with SearchStax Support to evaluate this situation.

The post JVM Heap Used appeared first on SearchStax Docs.

When a Solr deployment runs out of disk space, it may simply crash. Also, much lower levels of disk usage (below half) can make backups impossible.

Large Index Size

Some indexes are static, but most grow over time. Eventually, a growing index will strain the disk space allocated to it. When that happens, we can upgrade the deployment or just add disk space.

Incidentally, replicas of the same index do not necessarily occupy the same amount of disk. See Nodes Use Different Disk Space for more depth on that subject.

Backup Issues

The Backup process is extremely disk-intensive. It can easily cause a Free Disk Space alert.

For more information about backups and disk space, see Why do Backups Fail? and How Much Free Disk Space Do I Have?

Add Disk Space

It is possible to add disk space to the nodes of a deployment by making a request of SearchStax Support. We can add disk space in blocks of 50GB for a small monthly fee.

The post Free Disk Space appeared first on SearchStax Docs.