Monitoring and Logging

Make sure to review the previous Management and License Configuration page before proceeding with the steps below.

Status Reporting via syslog

In addition to recording logs to local log files, the license server is able to also send logs to a syslog server, for monitoring and alerting. You may configure the address of the syslog server either in the configuration file or on the Settings in the web UI:

Enter the address of the syslog server used to monitor the license server; the license server will then also send log messages to the syslog server. Configure your syslog server to notify you in case of errors or warnings. See the Messages section below for details on the messages that you may see in the logs.

NOTE 1: You must use the address and port of the syslog server. Using the address without specifying the port will not work. You may also specify a transport prefix: udp:// for UDP or tcp:// for TCP. If the transport prefix is not specified, it will default to UDP.

NOTE 2: In v1.1.5, you must restart the license server after changes to the SyslogServerAddress field before the change will take effect.

Monitoring Another License Server

License Server monitoring allows you to detect conditions where a primary and backup license servers may have been accidentally configured with different sets of licenses. Without monitoring, this state could go undetected until one of the server needs to take over for the other one.

To enable license server monitoring, configure each server to monitor the other one. This is done through the BackupServerAddress in the configuration file, or by entering the address of the other server on the Settings page:

Enter the service address of the license server to monitor. Note that this is a passive monitoring option; no licenses are used or transferred. Each configured server checks that the other server has the same licenses and will generate an error if this is not the case. You will also get a warning if the other server fails to respond to queries, allowing you to detect network issues, or computer crashes, etc. as they happen.

NOTE 1: You must use the full address of the other license server in this field, including the port number. For example, if the other license server is deployed with IP address of 10.0.1.14, you would enter 10.0.1.14:42131 in the BackupServerAddress field.

NOTE 2: In v1.1.5, you must restart the license server after changes to the BackupServerAddress field before the change will take effect.

Monitoring Messages

When monitoring the status of another server, these are the messages you may see reported to syslog:

Configuration match

07-18-2023	13:42:21	User.Info	127.0.0.1	Info: The configuration for license server 127.0.0.1:42132 now matches ours

This info message indicates that the licenses on the other server match the licenses on the current server.

Configuration mismatch

07-18-2023	13:44:21	User.Error	127.0.0.1	Error: License server 127.0.0.1:42132 configuration mismatch!

This error message indicates that some of the licenses, or license parameters are not identical between the two servers.

Other server fails to respond

07-18-2023	13:47:11	User.Error	127.0.0.1	Error: No reply from license server at 127.0.0.1:42132

This error message indicates that the network connection to the other server is down or that the other system is not responding.

NOTE 1: Warning and error messages are reported only once. Make sure that your syslog server alerts you of error conditions right away.

NOTE 2: It is normal and expected that you will receive some error messages while you are in the process of updating the licenses on the servers, since the license information will temporarily be different. However, once the servers are configured with the same licenses, you should see a "configuration match" message.

Product Monitoring

The license server monitors and reports the state of each Telos Alliance product that uses the license server. When a product is first detected, the server will report this message to syslog:

07-18-2023	13:53:01	User.Info	127.0.0.1	New product from 10.0.1.121:51770: {"ver":1,"op":"report","count":1,"servers":["10.0.1.11:42131","10.0.1.12:42131"],"features":{"channels":21}}

The message contains a list of servers configured in the product and a sum of the license features used by the product. The servers order is significant: the first server in the list is the server that the product is using for license validation. If the server order changes in a subsequent message, it indicates that the product switched servers, perhaps due to the primary server not responding.

You will see a new message any time a product change is detected. You will also get a warning if a product stops reporting its status to the server:

07-18-2023	14:01:45	User.Warning	127.0.0.1	Warning: Product at 10.0.1.121:51770 hasn't reported for 15 seconds

Server Liveness Monitoring

While operating, the license server will report "Alive" messages to the syslog server every 15 seconds:

07-18-2023	14:02:10	User.Info	127.0.0.1	Alive 95: 3 requests, 0 read errors, 0 send errors
07-18-2023	14:02:25	User.Info	127.0.0.1	Alive 96: 3 requests, 0 read errors, 0 send errors
07-18-2023	14:02:40	User.Info	127.0.0.1	Alive 97: 3 requests, 0 read errors, 0 send errors

The message contains a counter, incremented with each message, along with the number of license requests and read/send errors. Note that the error counters are informational and occasional errors are not a cause of concern; the system will just retry in case of an occasional error. If the error counters are rapidly incrementing, however, further investigation may be warranted.

Log Messages

The following log messages are written to the logs and may also be sent to a syslog server, if one is being used.

Info Message

"Loading license file '%s'...", fullFileName "Added license '%s'...", licenseKey "Loading license file '%s'...", fullFileName "Info: %s: add '%s'", notFoundRedirectFile, path "Info: %s not found: using default behavior", notFoundRedirectFile "Starting HTTP server on ", srvAddr "Error Starting the HTTP Server :", err "dispatch: ", err "Error listening:", err.Error() "Listening for application connections on", listenAddress "Error accepting: ", err.Error() "New product from %s: %s", fromAddr, string(data) "Request not recognized: '%s'", string(msgData) "Info: The configuration for license server %s now matches ours", addr.String() "Configuration was loaded from '%s'...", cfg.file "Logs folder : '%s'", cfg.data.LogsFolder "Licenses folder: '%s'", cfg.data.LicensesFolder "Service address: '%s'", cfg.data.ServiceAddress "CTRL-C detected, exiting licsrv..." "Running server, the user confirmation code is '%s'", cfg.data.ConfirmationCode

Warning Messages

"Warning: Unable to load default config, using defaults. ", err "Warning: '%s' not found", name "Warning: License servers changed: %s", string(data) "Warning: License features changed: %s", string(data) "Warning: Product at %s hasn't reported for %d seconds", addr, interval/time.Second "Warning: Error read from UDP socket: %s", err.Error() "Warning: Message %s from %s was not recognized: %s", msg.Op, addr.String(), string(msgData)

Error Messages

"Unable to load license file '%s': %s", fullFileName, err2.Error() "Failed to add license '%s'", licenseKey "Unable to load license file '%s': %s", fullFileName, err2.Error() "Error: Unable to generate license request" "Error: Unable to write go license request to", reqfile, ",", err.Error() "Error: Failed to parse JSON message: '%s'", string(data) "Error: Unable to convert message: %s", err.Error() "Error: Unable to resolve server address %s: %s", otherLicenseServer, err.Error() "Error: Unable to bind to UDP port %s: %s", udpListenAddress, err.Error() "Error: No reply from license server at %s", otherLicenseServer "Error: Unable to retrieve database hash! %s", err.Error() "Error: Hash message was not recognized: %s", err.Error() "Error: License server %s configuration mismatch!", addr.String() "Error: Failed to load config.json from", cfg.file, ":", err.Error() "Failed to load license files" "Failed to create license repository"