Introduction

OpenStack is a complex suite of software that can make tracking down issues and faults quite daunting to beginners and experienced system administrators alike. While there is no single approach to troubleshooting systems, understanding where OpenStack logs vital information or what tools are available to help track down bugs will help resolve issues we may encounter.

Checking OpenStack Compute Services

OpenStack provides tools to check various parts of Compute Services, and we’ll use common system commands to check whether our environment is running as expected.

Getting ready

To check our OpenStack Compute host we must log in to that server, so do this now before following the given steps.

How to do it…

To check that Nova is running the required services, we invoke the nova-manage tool and ask it various questions of the environment as follows:

• To check the OpenStack Compute hosts are running OK:

  sudo nova-manage service list

  You will see the following output. The 🙂 icons are indicative that everything is fine.

  

• If Nova has a problem:
  If you see XXX where the 🙂 icon should be, then you have a problem.

  

  Troubleshooting is covered at the end of the book, but if you do see XXX then the answer will be in the logs at /var/log/nova/.

  If you get intermittent XXX and 🙂 icons for a service, first check if the clocks are in sync.

• Checking Glance:
  Glance doesn’t have a tool to check, so we can use some system commands instead.

  ps -ef | grep glance
  netstat -ant | grep 9292.*LISTEN

  These should return process information for Glance to show it is running and 9292 is the default port that should be open in the LISTEN mode on your server ready for use.

• Other services that you should check:
  • rabbitmq:

    sudo rabbitmqctl status

    The following is an example output from rabbitmqctl when everything is running OK:

    

  • ntp ( N etwork Time Protocol, for keeping nodes in sync):

    ntpq -p

    It should return output regarding contacting NTP servers, for example:

    

  • MySQL Database Server:

    MYSQL_PASS=openstack
    mysqladmin -uroot –p$MYSQL_PASS status

    This will return some statistics about MySQL, if it is running:

    

How it works…

We have used some basic commands that communicate with OpenStack Compute and other services to show they are running. This elementary level of troubleshooting ensures you have the system running as expected.

Understanding logging

Logging is important in all computer systems, but the more complex the system, the more you rely on being able to spot problems to cut down on troubleshooting time. Understanding logging in OpenStack is important to ensure your environment is healthy and is able to submit relevant log entries back to the community to help fix bugs.

Getting ready

Log in as the root user onto the appropriate servers where the OpenStack services are installed.

How to do it…

OpenStack produces a large number of logs that help troubleshoot our OpenStack installations. The following details outline where these services write their logs.

OpenStack Compute Services Logs

Logs for the OpenStack Compute services are written to /var/log/nova/, which is owned by the nova user, by default. To read these, log in as the root user. The following is a list of services and their corresponding logs:

• nova-compute: /var/log/nova/nova-compute.log
  Log entries regarding the spinning up and running of the instances
• nova-network: /var/log/nova/nova-network.log
  Log entries regarding network state, assignment, routing, and security groups
• nova-manage: /var/log/nova/nova-manage.log
  Log entries produced when running the nova-manage command
• nova-scheduler: /var/log/nova/nova-scheduler.log
  Log entries pertaining to the scheduler, its assignment of tasks to nodes, and messages from the queue
• nova-objectstore: /var/log/nova/nova-objectstore.log
  Log entries regarding the images
• nova-api: /var/log/nova/nova-api.log
  Log entries regarding user interaction with OpenStack as well as messages regarding interaction with other components of OpenStack
• nova-cert: /var/log/nova/nova-cert.log
  Entries regarding the nova-cert process
• nova-console: /var/log/nova/nova-console.log
  Details about the nova-console VNC service
• nova-consoleauth: /var/log/nova/nova-consoleauth.log
  Authentication details related to the nova-console service
• nova-dhcpbridge: /var/log/nova/nova-dhcpbridge.log
  Network information regarding the dhcpbridge service

OpenStack Dashboard logs

OpenStack Dashboard (Horizon) is a web application that runs through Apache by default, so any errors and access details will be in the Apache logs. These can be found in /var/log/ apache2/*.log, which will help you understand who is accessing the service as well as the report on any errors seen with the service.

OpenStack Storage logs

OpenStack Storage (Swift) writes logs to syslog by default. On an Ubuntu system, these can be viewed in /var/log/syslog. On other systems, these might be available at /var/log/messages.

Logging can be adjusted to allow for these messages to be filtered in syslog using the log_level, log_facility, and log_message options. Each service allows you to set the following:

If you change any of these options, you will need to restart that service to pick up the change.

Log-level settings in OpenStack Compute services

Many OpenStack services allow you to control the chatter in the logs by setting different log output settings. Some services, though, tend to produce a lot of DEBUG noise by default.

This is controlled within the configuration files for that service. For example, the Glance Registry service has the following settings in its configuration files:

Moreover, many services are adopting this facility. In production, you would set debug to False and optionally keep a fairly high level of INFO requests being produced, which may help with the general health reports of your OpenStack environment.

How it works…

Logging is an important activity in any software, and OpenStack is no different. It allows an administrator to track down problematic activity that can be used in conjunction with the community to help provide a solution. Understanding where the services log, and managing those logs to allow someone to identify problems quickly and easily, are important.