Monitoring Looker

On this Page
Docs Menu

Although Looker application monitoring may not seem like it is strictly required, it is very important to setup. In the rare instance that something goes wrong with your server, it is often much more difficult or impossible for Looker to help you fix the issue unless you can provide appropriate monitoring information from the time of the incident.

Application Monitoring

URL

If you need a simple way to validate that your Looker instance is running, you may append /alive to your Looker instance’s URL.

For example: https://my.looker.com/alive

If your instance is up and running you’ll receive a 200 OK http status code.

JMX

The Java virtual machine which runs Looker may be monitored via JMX.

Many monitoring applications such as Zabbix and Nagios support JMX. Please see your monitoring application’s documentation for more information.

Edit the Looker Startup Script

To enable JMX monitoring, you will need to edit your Looker startup script. By default it is named:

/home/looker/looker/looker

Look for the java startup parameters:

java \
  -XX:+UseG1GC -XX:MaxGCPauseMillis=2000 \
  -Xms$JAVAMEM -Xmx$JAVAMEM \
  -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps \
  -Xloggc:/tmp/gc.log  ${JAVAARGS} \
  -jar looker.jar start ${LOOKERARGS}

Add a section following the line starting with -Xms$JAVAMEM:

  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote \
  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote.port=9910 \
  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote.ssl=false \
  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote.local.only=false \
  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote.access.file=${HOME}/.lookerjmx/jmxremote.access \
  -Dcom.sun.akuma.jvmarg.com.sun.management.jmxremote.password.file=${HOME}/.lookerjmx/jmxremote.password \

Create the .lookerjmx directory

Next, create the .lookerjmx directory under your looker user’s home directory, and set permissions:

sudo su - looker
mkdir ~/.lookerjmx
chmod 700 ~/.lookerjmx
cd ~/.lookerjmx

Create the JMX Files

Using your favorite text editor create a file in the new directory named jmxremote.access with the following contents (you may customize for your environment):

monitorRole   readonly
controlRole   readwrite \
              create javax.management.monitor.*,javax.management.timer.* \
              unregister

Next create a file named jmxremote.password in the same directory with the following contents, using your own secure passwords:

monitorRole   some_password_here
controlRole   some_password_here

Setting Permissions

Make it such that Java (and therefore Looker) will not start if the file permissions allow anyone except the looker user to read the password file.

chmod 400 jmxremote.*

Restart Looker

Looker needs to be restarted to enable JMX. Make sure you run this as the looker user and NOT root:

cd ~/looker
./looker restart

Your Looker instance is now configured for remote JMX monitoring on port 9910, using the password you supplied. You may need to modify your firewall settings or network ACLs to allow your monitoring server to get network access on this port.

Host Monitoring

For every host running the Looker application, we recommend that you collect, graph and alert on at least the following performance metrics:

  • CPU Utilization: load and percent CPU utilized
  • Memory Utilization: total used and swap used
  • Disk Usage

Alerting Thresholds

To establish good alerting thresholds first establish a baseline. Collect performance data with your Looker instance running under a normal load. Take a look at the performance graphs and observe the peaks. The length of time you will need to establish the baselines depends on your business and your Looker usage patterns. Some companies may use Looker in a stable, repeatable pattern every week during business hours. Others may use Looker more heavily at specific time periods (such as the end of each month).

In general, alerts should only be sent for events which are actionable. Sending alerts when there is nothing which needs to be done masks the importance of critical alerts.

The following thresholds may be used as a starting point for alerting. When the following values are exceeded for 15 minutes or more, manual intervention may be required.

Metric Warning Critical Comments
CPU Load 2 4 Load should generally be 1 or less for a single core system. Sustained high load leads to poor performance
CPU % Used 80 90 High CPU usage leads to poor performance
Memory % Used 60 70 High memory usage can indicate too much memory is allocated to Java
Disk % Used 80 90 Ensure the disk isn’t full

Additional notes:

  • Systems with more than one core can handle high CPU loads without reduced performance. The rule of thumb is that sustained load should not be greater than the number of processor cores.
  • The percent of total CPU time in use before a system experiences performance degradation scales with the number of CPU cores in the system. In other words, a single core system may have poor performance when the CPU is 80% utilized, whereas a sixteen core host may still be usable at 95% utilization.
  • High sustained CPU utilization can be rectified by upgrading the host hardware, or upgrading to a larger instance. Sometimes large numbers of scheduled looks or long-query derived tables can be reduced or made more efficient to improve performance.

Next Step

After you have setup monitoring you’re ready to setup Looker backups.

Still have questions?
Go to Discourse - or - Email Support
Top