Pathfinder Core PRO provides a variety of different logging mechanisms as well as maintaining its own internal system logs. There are two kinds of log files created by the system: System Logs and User Logs.
System Logs are created by the system's internal services and are mainly for troubleshooting with Telos support.
User Logs are where you can define changes on your Axia network that you want to be entered into log files for later review.
Click on the System > Logs link in the Navigation bar to view the current logs in the system as well as to define logging options. The page opens on the User Logs; use the System Logs and User Logs links at the top to switch between the two.
The page has three parts:
Time range, which applies to everything below it
the tabs Files, Search, Summary, Timeline and Saved searches
Delete/Clear All Logs, at the bottom, which deletes or clears every log in the list after confirming
Important Note: The log tools are for administrators. Other users, or any user while the log service is unavailable, see a plain list of log files.
Time Range
Pick Last 15 min, Last hour, Last 24 h, Last 7 days or All, or type a from and to time and click Apply. All times are the unit's own time (unit time), whatever time zone your browser is in.
The chart below the buttons shows how many log entries were written over the range. Red marks errors and yellow marks warnings. Drag across the chart to narrow the range to that stretch of time.
Files
The Files tab lists every log:
Column | Description |
|---|---|
Log | The Filename of the log. A log that has been rotated shows how many files it has; click the arrow beside it to list them. See Log File Maintenance later in this article. Clicking a log Filename opens the log in a browser. |
Covers | The time of the first and last entries. |
Size | The size of the log, all its files together. |
Last updated | The date and time the log was Last Updated. |
Last 24 h | The number of errors and warnings in the last 24 hours, or quiet if there were none. Click one to step through them on the Timeline tab. |
Each log has these buttons:
View opens the log in the page, as columns (time, writer, operation, object and properties), with warnings and errors colored. Use Older and Newest to move through it, Follow to keep showing new entries as they are written, Open file to open the raw file, Download to save it, and Close to close the viewer.
Tail shows the last lines of the log as plain text and keeps adding new lines as they are written. Set the number of Lines and click Tail to start again, or Stop to stop following.
Download saves the log. For a rotated log, the files are joined, oldest first, into one download; each file can also be downloaded on its own from the list of files.
Delete/Clear deletes the log's files, or empties the file that is being written to.
The system logs list looks the same, with the logs of the system's services and the operating system:
Benign Warnings and Errors
Some warnings and errors are expected on a particular system and are not a sign of trouble, for example a device that is switched off on purpose. On the Search and Timeline tabs, click Mark benign beside such an entry. Entries like it are then left out of the warning and error counts, so the counts show what needs attention. Tick Count benign on the Files tab to include them again.
Search
The Search tab finds entries in any number of logs, including their rotated files, within the time range.
Choose Contains or Does not contain, type the text, and choose how it is matched: anywhere in the entry, as a whole word, or as a regular expression. Tick Match case to match upper and lower case exactly.
Click + Add criterion to add more terms, and choose whether Entries must match all of the included terms or any of them.
Show limits the results to errors, warnings and information entries.
In: narrows the search to particular logs; by default every user and system log is searched.
Context shows none, 1 or 3 entries before and after each match.
Click Search. Each match shows the log it is in, with View in log to open the viewer at that entry and Around this time to open the Timeline at that moment. Click Export .txt to save the results as a text file.
Saved Searches
Click Save search… to keep a search for later.
The Saved searches tab lists them, with the number of new matches since each was last looked at. The same number appears on the tab itself whenever the Logs page opens, so a saved search works as a simple watch for a problem coming back.
Click Run to run a search again, Rename… to rename it, or Delete to remove it. Saved searches are shared by everyone who uses the Logs page.
Summary
The Summary tab summarizes the user or system logs over the time range. Pick User logs or System logs and click Summarize.
The summary shows the number of entries, warnings, errors, and starts and restarts of the system's services, followed by:
Most repeated warnings and errors, grouped by message, with Find to search for them and Timeline to see when they happened
Devices that dropped, REST endpoints failing, and Starts and restarts
Click Download support bundle to save a ZIP file to send to Telos support. It holds the logs cut to the time range, the summary, and a description of the unit. Keys in REST addresses are removed from it.
Timeline
The Timeline tab shows every log's entries around a moment. Pick a time in Around, how far either side to look (for example ± 5 min), and which logs, then click Show. You can also get here from Around this time on a search result, or by clicking a log's error or warning count.
Each log is a lane, with a mark for each entry: blue for information, yellow for warnings and red for errors. Below the lanes, the entries are listed in time order. When you arrive from an error or warning count, Previous and Next step through those entries one at a time.
Creating New Log Writers
To see the list of Log Writers that have been created in the system, click the Log Writers link on the Logs page.
Each Log Writer in the system will be displayed on this page by Name along with its type and location.
To create a new Log Writer, click on Add + on the Log Writers screen.
Type a Name for this Log Writer in the Log Writer Name field.
From the Log Writer Type drop-down list, select this Log Writer’s Type. There are four types of Log Writer that can be created:
Log Writer Type | Description |
|---|---|
Log File | Stored locally on the Pathfinder Core PRO system, these logs are subject to regular rotation, deletion, and cleanup depending on space requirements. This Log File Type is useful for logging recent events but should not be relied upon for long term storage. |
TCP Listener | This type of Log Writer will listen on a TCP port and send the log entries as plain text messages to any application that connects. For example, you could use a putty (telnet) session to connect to the system and monitor log changes dynamically. |
TCP Client | This type of Log Writer will attempt to connect to an IP address and port, and if it can make the connection, will send the log messages as plain text messages to the listening application. |
Syslog | This is the recommended method for long term log storage. Syslog is a highly utilized standard protocol for log collection, viewing, and storage used by many systems and IT scenarios. A number of syslog collection applications exist, both freeware and commercial. A quick internet search should provide you with a variety of options. |
Fill in the fields for the type you chose:
Type | Fields |
|---|---|
Log File | None. The file is named after the Log Writer, with .log added. |
TCP Listener | Type the Port associated with the TCP application. Also Max Offline Cache Count (how many entries, 1–5000, are kept for an application that connects later) and Connect Get Requests. |
TCP Client | Type the IP Address and Port associated with the TCP application. Also Max Offline Cache Count and Connect Get Requests. |
Syslog | Type the IP Address of the collector application. |
Tick Skip Web Client SapV2 to leave out the SapV2 messages exchanged between the web pages and the system itself (from 127.0.0.1). It only matters when a SapV2 option under MessageLogging is ticked, and keeps that log to the traffic of outside applications.
Use the selection window at the bottom of this dialog to select what log options you wish to log. Expand the tree and check the items you wish to log.
The available items are listed below. The number beside each is its message type id, which appears in every log entry it writes (see Log Message Structure).
Group | Item (id) | What it logs |
|---|---|---|
AudioAlarms | AlarmState (3001) | Generates a log message whenever an audio alarm state changes. |
AudioAlarms | LvlState (3002) | Generates a log message whenever the audio threshold for silence is passed for longer than 250 ms. This occurs before the countdown to an alarm state begins. It can be used to log the actual transitions being reported by the equipment but may also generate a large amount of log content depending on the type of audio content being monitored. |
Clustering | ConsolidatedNodeStates (15001), IsPrimaryActive (15002), AllNodesAttempted (15003), SyncProgress (15004), NodeState (15005), EventSystemIsPrimary (15006), Online (15007) | Changes to the cluster's state and each node's state. |
ConnectMessage | Connected (1001) | Logs connections and disconnections from equipment. Use this log message to find equipment with which Pathfinder Core PRO is struggling to maintain a connection. |
ConnectMessage | Online (1002) | Logs changes to the online/offline state of the device. For a device to be considered online, proper communications must be occurring on all identified ports. |
ConnectMessage | LoginState (1003), LastConnectionState (1004) | A device's login state, and the reason its connection last changed. |
Custom | (80001–99999) | Any custom SAPv2 subscriptions that have been defined through the Add Custom button. Each must have a Name, an unused Log Type Id between 80001 and 99999, a Log Severity (Emergency, Alert, Critical, Error, Warning, Notice, Informational or Debug), and a valid SapV2 subscription message. |
DeviceEmulators | ConnectedCount (11001), Connected (11002) | Connections to device emulators. |
DeviceEmulators | WatcherTriggered (11003), WatcherTriggeredValue (11004) | Watchers that trigger, and the value that triggered them. |
DeviceEmulators | LastSent (11005), LastReceived (11006), GenericEmulatorsLastSent (11007), GenericEmulatorsLastReceived (11008), ProbelEmulatorsLastSent (11009), ProbelEmulatorsLastReceived (11010) | What emulators send and receive: all emulators, or only generic or only Probel ones. |
DeviceEmulators | ProbelEmulatorsRouteChangeRequestReceived (11011) | Route change requests received by Probel emulators. |
Devices | VMIXState (7001), VMIXGain (7002) | Logs changes to VMIXer On/Off settings and VMIXer gain settings. |
LegacyPanels | PanelPropertyChanged (5001) | Logs changes to properties of any of the controls in user panels. |
LogicFlows | TranslatorOutputChanged (4001) | Logs changes whenever a logic flow translator changes its output. |
LogicFlows | CombinerOutputChanged (4002) | Logs changes whenever a logic flow combiner changes its output. |
MemorySlots | MemorySlotChanged (6001) | Logs whenever a memory slot changes its value. |
MessageLogging | LwrpIncoming* (9001), LwrpOutgoing* (9002), LwcpIncoming* (9003), LwcpOutgoing* (9004) | Lwrp and Lwcp incoming and outgoing messages. |
MessageLogging | SapV2InternalIncoming* (9005), SapV2InternalOutgoing* (9006) | Incoming and outgoing SapV2 messages between services. |
MessageLogging | SapV2ExternalIncoming* (9007), SapV2ExternalOutgoing* (9008) | SapV2 messages coming from outside the system and being sent outside of the system. |
MessageLogging | LoginFailures (9009), LoginSuccesses (9010), AccessViolations (9011) | Logins, and attempts to use something the user has no rights to. |
MessageLogging | AuditGet (9012) | Log incoming SAPv2 messages that are requesting information for security audit purposes. |
MessageLogging | AuditSet (9013) | Log incoming SAPv2 messages that are requesting changes for security audit purposes. |
RestApi | LastSent (12001), ResponseData (12002), ResponseCode (12003), ResponseSuccess (12004) | Requests sent to REST API devices and their responses. |
RouterEvents | RouteState (2001) | Logs any route changes that take place within the system. |
RouterEvents | Gpio State (2002) | Logs changes to GPIOs within the system. |
SapPropertyRouter | DestinationMessageBeingSent (13001) | Messages sent to SAP property router destinations. |
Scenes | SceneIsActive (14001), SceneState (14002), ActivateSceneRequested (14003), ItemsAreActive (14004), ItemsCurrentValue (14005) | Scene activations and the state of their items. |
Scte35 | 10001–10025 | SCTE-35 cue messages received and triggered, their fields, and the state of the SCTE-35 reader. |
TimeEvents | Elapsed (8001) | Logs whenever timers elapse. |
TimeEvents | Enabled (8002) | Logs changes to a timer's enabled state. |
Items marked with an asterisk (all in MessageLogging) should be used with support or when trying to find a problem as these can increase CPU load. These are for troubleshooting only.
When all fields are complete, click Apply to save your changes and create the Log Writer.
Log File Maintenance
You may notice that some logs have multiple copies with a period and a number at the end. The Files tab shows these together under one log.
Due to space restrictions, Pathfinder Core PRO maintains an intelligent log rotation service which will rotate the log files as they become too large: the current file is renamed with .1 added, older files move up one number, and the oldest is deleted.
There are some other rules which this service uses that might generate more aggressive log manipulation and/or deletion if space is becoming limited. For this reason, it is recommended that you use an outside syslog-based logging service to capture logs that you wish to store for longer periods of time. Review the Creating New Log Writers section for additional details.
Log Message Structure
Pathfinder log messages follow one of three different formats depending on the log writer options selected and the type of message. If you choose Syslog as the log writer type, then the syslog structure is used. All other writer types use the standard log message structure. If the message type is one of the audit get or audit set message types the log message data is wrapped in an Audit log message structure.
Standard log message structure
By default, log messages are composed of a timestamp, a message type id number, and a sap message:
<TimeStamp> <MessageTypeId> <sap message>
For example:
12-26-2023_11:45:00.001 6001 indi MemorySlots#0.MemorySlot#Time3 SlotValue=FalseTimestamp: Date and Time structured as: MM-dd-yyyy_HH:mm:ss.fff
MessageTypeId: Each message item in the log writer editor checkboxes has an associated id number (listed above). So for example memory slot value changes have an id of 6001. This allows for easy filtering of specific message types.
Message: Each message is a SapMessage comprised of an operator, object path, list of properties, and in some cases a list of system items. For more details on the structure of SapMessages see Appendix A.
Note: two spaces are used between the timestamp and the message type id and again between the message type id and the sap message data itself.
Audit log message structure
Audit log messages follow the same format as standard log messages except that the SapMessage wraps the original message with additional data:
01-03-2024_13:59:47.554 9012 indi AuditGet#[tcp://192.168.1.230:41202/] Direction=Incoming, Message="ClusterAdmin:get Devices#0 Ping<CR,LF>"
01-03-2024_13:59:49.230 9012 indi AuditGet#[tcp://192.168.1.230:9600/] Direction=Incoming, Message="<NoUser>:indi Devices#0 Ping=Pong<CR,LF>"An audit log message will include an object path of AuditGet or AuditSet with a URL where the original audit message originated. Typically the url will begin with either tcp:// or ws:// for a tcp or websocket connection followed by the IP and Port. The Direction property indicates the direction of the message as incoming or outgoing. The message property itself will contain the user name used followed by a colon and the original SapMessage. Outgoing messages may specify <NoUser> as the user since the outgoing message (especially if it is a change message) is not necessarily associated with a specific user. Carriage returns and line feeds in the original message are expressed as <CR,LF>.
Syslog message structure
When syslog is chosen for the log writer in Pathfinder, RFC 3164 format is used except that Pathfinder's time stamp portion of the message also includes the milliseconds.
<<severity>><timestamp> <localip> PFC: <MessageTypeId> <MessageObjectPath> <PropertyList>
<14>Jan 03 16:15:02.619 192.168.1.96 PFC: 6001 MemorySlots#0.MemorySlot#ttt SlotValue=B
<14>Jan 03 16:22:11.150 192.168.1.96 PFC: 9012 AuditGet#[ws://[::1]:56483/] Direction=Incoming Message="Admin:GET Devices#0 Ping<CR,LF>"A syslog message is made up of a severity calculation followed by a date time stamp in the format: MMM dd HH:mm:ss.fff (normally RFC 3164 does not include the millisecond portion). Next comes a single space and the message portion which includes a tag (usually process name) and then the message data. We use PFC: for the tag for Pathfinder Core PRO. The message itself is the object path and property list from the SapMessage as described in the standard and audit logging above. The operator and systems items from the SapMessage are not included when syslog options are selected.