(For more resources related to this topic, see here.)

Monitoring a live system is crucial to maintaining stability and performance; not only to avoid potential failures but even for debugging and tracing back an event. That is why having a system record its activities results in a rich database of logs that can be used for investigation. Logfiles tell a fascinating story to those who can read it. They carry the history of all events narrated in thorough detail. ArcGIS for Server, like any other system, keeps logfiles for all events, from the basic “connection established” event to the severe “service failed to initiate” event.

Logging levels

Recording events on Server is done at different levels. You can tell Server to log every single event as it happens or filter to record only fatal errors. Consequently, recording fine events generates more logs than recording only those messages with errors. There are seven logging levels, and these are described in the following Esri table:

Log level	Description
Severe	This level logs serious problems that require immediate attention. It only includes severe messages.
Warning	This level logs moderate problems that require attention. It also includes severe-level messages.
Info	This level logs common administrative messages from Server, including messages about service creation and startup. It also includes severe and warning messages.
Fine	This level logs common messages from users of Server, such as names of operation requests received. It includes severe, warning, and info messages.
Verbose	This level logs messages providing more details about how Server completes an operation, such as whether each layer in a map service was drawn successfully, how fast the layer was drawn, and how long it took Server to access the layer’s source data. This level includes severe, warning, info, and fine messages.
Debug	This level logs highly verbose messages designed for developers and support technicians who want to obtain a better understanding of the state of Server when troubleshooting. This level should not be used in a production environment as it may cause a significant decrease in Server performance.
Off	At this level, logging is turned off. Events are not logged with Server.

As you can see, Debug is the finest level and keeps Server busy with logging events, making other important tasks suffer.

Log analysis

Logs can be viewed and refreshed actively from the ArcGIS for Server Manager window as they are written. To see your current logfiles, go to Manager and activate the Logs tab:

Naturally, each GIS server generates its own logs and this is all saved by default at C:arcgisserverlogs. You cannot use a shared folder for this; each GIS server should generate its own logs in its directory, ArcGIS for Server aggregates those logs into the Server site, in a table view with filters options, which allows you to search through the logs. From the View Log Messages panel, click on Query to view the current logfiles as shown in the following screenshot. You might get messages different from mine.

You might not have any messages if your current log level is set to only record errors, and there are no errors. To change the log level, click on Settings. From the Log Settings form, select Verbose from the Log Level drop-down list. You can set the logs to be cleared automatically if you want to. Keep the rest of the settings intact and click on the Save button.

By default, the logs are kept on the GIS server for three months. If you are planning to keep the logs for longer than that, perhaps for offline analysis, you may want to archive them periodically and delete them. Generally, clearing the logs is better for performance. This will be discussed in the coming pages.

Best practice

Since logs are saved to disk frequently, they use high IO. It is recommended that you point the log path to a local directory, preferably on a Solid State Drive (SSD) for best performance.

Now, let us see how the logs are being generated. First of all, let us clear all the logs to start afresh. To do that, click on Delete Logs from ArcGIS for Server Manager and then click on Yes, as shown in the following screenshot:

Now that the logs are cleared, we will activate the parcels service by simply visiting the REST URL and then checking the log.

Type the REST URL on a new browser page and press Enter. You should see something like the following screenshot if you have access to the service:

Go back to your logs and click on Query to refresh the page. You should see one message in the table. You might see other messages from Server that happened to be executed at that particular time but look for this one:

Level	Time	Message	Source	Machine	User Name
INFO	Nov 17, 2013, 11:18:26 AM	Request user: Anonymous user, Service: Parcels/MapServer	Rest	GIS-SERVER01	Anonymous user

The level is INFO, which means a detailed event; it says a request user from REST consumed the Parcels Map Service and GIS-SERVER01 served that request. If you have security enabled, you would even know which user consumed that service. Now, let us take it to the next level. On the Parcels REST page, click on ArcGIS JavaScript to view the map with the service loaded.

Go back to the log view and click on Query to refresh; make sure the Log Filter dropdown is set to Verbose. A fleet of messages was generated from our last action; we will take a look at each line and analyze it.

There are many columns that can be displayed on the log table and you can show or hide them from the Columns button.

For a better view, you can click on the Printer Friendly View link, which will display a text format version of this table in a new page.

This is the log we are going to analyze; we will start from the first line:

INFO, Nov 17, 2013, 11:29:17 AM, Request user: Anonymous user, 
  Service: Parcels/MapServer Rest.

This is a request to consume the service. You can use this identifier to measure how many times a service has been requested.

FINE Nov 17, 2013, 11:29:17 AM REST request received. Request size 
  is 178 characters. Parcels.MapServer

The preceding line is appended if there is more work to be followed; it shows the request size in bytes.

FINE Nov 17, 2013, 11:29:17 AM Begin ExportMapImage 
  Parcels.MapServer

The process is so fast that we are still in the same second. The preceding line of code tells us that the Export Map Image process just started. This is the big process where Server exports an image of the desired area; however, there is still more work to follow to create the actual image. You can start measuring the drawing time of a certain service from this line.

VERBOSE Nov 17, 2013, 11:29:17 AM Begining of preparation. 
  Parcels.MapServer
VERBOSE Nov 17, 2013, 11:29:17 AM End of preparation. 
  Parcels.MapServer

The preceding two lines highlight the preparation of the export image process. They usually happen very fast.

FINE Nov 17, 2013, 11:29:17 AM 
  Extent:1467314.863829,2191233.084700,2574598.328396,2702665.79038;
    Size:1574,727; Scale:2658831.00Parcels.MapServer

A map needs an initial extent coordinates for it to draw. At the first call of the service, Server implicitly sends the default full extent to draw the map. After that, the user will explicitly request a new extent, each time he/she zooms in or pans the map.

VERBOSE Nov 17, 2013, 11:29:17 AM Beginning of layer draw: Parcels 
  Parcels.MapServer

Since we only have one layer, you will see one occurrence of this line; however, you will see these lines reappear with more layers and there will be more logs to follow.

VERBOSE Nov 17, 2013, 11:29:17 AM Execute Query Parcels.MapServer

I consider this one of the most important lines; this is where the database is advised and queried to get the actual features. You can make a good measurement here by monitoring how long an execute query takes. If this takes a long time to execute, you might want to consult your DBA to look into tuning the database.

VERBOSE Nov 17, 2013, 11:29:17 AM Symbol Drawing Parcels.MapServer
VERBOSE Nov 17, 2013, 11:29:17 AM Data Access Parcels.MapServer
VERBOSE Nov 17, 2013, 11:29:17 AM Symbolizing Parcels.MapServer

Symbology work, depending on the user, can be executed either on the server or on the client. Since we are running on a browser, the symbology drawing will be carried on the client’s browser by JavaScript. Note that this is only the symbology drawing; the labeling is done in another step.

VERBOSE Nov 17, 2013, 11:29:17 AM Number of features drawn: 10
  Parcels.MapServer

This message shows the number of features that have been drawn. This line is useful if you want to know how many features are retrieved for each request and monitor the performance.

VERBOSE Nov 17, 2013, 11:29:17 AM End of layer draw: Parcels 
  Parcels.MapServer

This line signifies the end of layer drawing; you should now start seeing the map, but with no labels.

VERBOSE Nov 17, 2013, 11:29:17 AM Beginning of labeling phase 
  (labeling and label draw) Parcels.MapServer

Now that the symbology work is done, the labeling will start. This will give you even more measurement indicators for performance.

VERBOSE Nov 17, 2013, 11:29:17 AM Symbol Drawing Parcels.MapServer

It draws the font symbol as described in the layer description which can be found in the layer properties.

VERBOSE Nov 17, 2013, 11:29:17 AM Number of features drawn: 10 
  Parcels.MapServer

The preceding line indicates that the features have been labeled successfully.

VERBOSE Nov 17, 2013, 11:29:17 AM End of labeling phase (labeling 
  and label draw) Parcels.MapServer

The preceding line marks the end of the labeling phase.

FINE Nov 17, 2013, 11:29:17 AM End ExportMapImage 
  Parcels.MapServer

The map image has been exported successfully; we will attempt to deliver it to the client after this.

FINE Nov 17, 2013, 11:29:17 AM REST request successfully 
  processed. Response size is 6364 characters. Parcels.MapServer

The last message describes the response map, which is a 6K map image. My Server is so fast that the whole thing happened in the same second. This is not much by way of a log analysis. However, in the next topic we will attempt to analyze a much richer log and will try to answer some questions.