Health Monitoring
Macula provides health monitoring data for servers, devices and channels, stream recording, as well as live status of connected user sessions and other resource information, including external services. In the Archive statistics section, it is possible to view the stream details and also un-protect footage that has been locked via Macula Monitor application.
To access live reports in Macula Console, choose the Monitoring section in the bottom-left-hand panel and switch between components using the menu on the left. Use the Search field in the upper-right-hand menu to filter the records; press the Refresh button or F5 on your keyboard to reload the item list. The statistics are not refreshed automatically, so do not forget to reload the list tin order to obtain the most recent information.
The contents of any subsection in Monitoring can be exported in CSV (comma-separated value) format using the button on the upper panel.
The Details button (available for some of the sections) on the top panel allows you to view detailed information about important events related to the target resource. For example, you can view the list of recent connection and VCA license errors for every channel. The details are discarded after each server reboot.
Each section here contains a number of columns with miscellaneous information. Not all of them are available by default; to modify the displayed details and their layout, use the grid icon 
in the upper right corner: an additional configuration window will pop up. You can hide the fields you do not need, and also freeze the leftmost columns for convenient horizontal scrolling. Hidden fields cannot be used for search or sorting.
All information written in plain black is informative. When things go wrong, there are two levels of errors here: critical and warning. Critical errors are marked red; those usually indicate important stuff like missing main stream video, no connection to services etc. Warnings are yellow and are used to highlight minor issues like missing secondary stream, too long GOP period, or not enough space on the disk. You can ignore warnings by right-clicking the selected items and selecting Ignore > (preference). This helps if the warning is irrelevant, e.g., if you do not use substreams. Critical errors cannot be ignored as they need to be fixed for correct server operation.
Further in this chapter you will find a brief description of each information field.
Servers
This section includes information about connection and synchronization between the central server and the rest of the servers, as well as each server status and system load.
For each server, the following information is displayed:
Title: server name
Status: current server status (Critical status, indicating an unreachable or otherwise "bad" server, is also marked with red with the corresponding error description in the Synchronization column)
Connection: connectivity status
Synchronization: configuration and audit sync status, including errors (if present)
Configuration: indicates if the central server has successfully applied current configuration to the connected server
Status update time: last sync/connection attempt timestamp
Version: server software version (version differences may cause sync issues!)
Failover cluster: failover cluster name, if applicable
Failover status: if applicable
Not in cluster: target server is not a part of any failover cluster (may be a mirror server, the central server itself, or a recording server not belonging to any clusters)
Normal: target server is a part of a failover cluster
Unknown: undefined; this status may appear in case of incompatible database version or corrupt configuration
Failover server: current failover node that is acting instead of the target server; if none, server is operating on its own (for regular recording servers only)
Failover configuration: original recording server configuration assigned to the target failover node; if none, failover node is idle (only for failover servers)
Connected/disconnected time: status change timestamp
Configuration load duration: how long it took for the server to load the configuration on startup
CPU load: CPU utilization of the target server, %
Process uptime: amount of time elapsed since the last service (re-)start, DD.HH:MM:ss
Process CPU load: CPU load from Macula service on the target server, %
Memory load: current memory utilization of the target server, %
Total physical memory: amount of RAM installed on the target server, GBytes
Available physical memory: amount of free physical memory on the target server, GBytes
Process virtual memory: amount of virtual memory used by Macula service, MBytes
Process physical memory: amount of physical memory used by Macula service, MBytes
Network load: maximum network interface load of the target server (i.e., if the server has multiple interfaces, utilization of the most loaded one will be shown), %
Network transfer rate: total bandwidth utilization of the target server (if server has multiple interfaces, aggregated bandwidth from all interfaces is displayed), MBytes per second
Free system disk space: total amount of free space left on the disk, where OS is installed (system disk, usually C:)
Information: any additional information, if any, will be displayed here
Machine load statistics are updated every five seconds but are not refreshed automatically when the Monitoring section is open; press F5 on your keyboard to refresh the section contents.
Note that all the columns will be present for all servers but for some server types some fields may be not applicable and will be therefore always empty: e.g., failover related details are not shown for the central server and its mirror, as well as for the recording servers that are outside failover clusters.
To verify the connection quality between Macula Global and any Macula Recording Server, go to that server settings in the Configuration tab, choose Connections tab in the server settings, and scroll to the very bottom.
For any Macula Recording Server, you can test the connection with the target Macula Recording Server by using the Test button in the bottom of this tab. This comes useful when:
you are adding the server without autodiscovery and helps you avoid issues like port mismatch, but mostly when
you need to verify the connection quality, resolve synchronization issues, connection drops, or low/unstable speed
The test is performed using different size packets (including large) and helps troubleshoot network issues when the connection can be established and maintained with small packets but larger ones are dropped).
Devices
The device summary provides configuration update status and the time of the last communication between the server and device, as well as shows, to which servers the devices are attached.
The following fields are available:
Title: device name
Status: current device condition
Normal: the device is functioning properly
Unknown: the device's channel is disabled, or the device core has not been loaded yet (may appear shortly after server startup)
Server: current server that has this device assigned (may be regular recording server or failover)
Devices/model: device type or model
Status update time: last sync/connection attempt timestamp
Information: additional information, if any, will be displayed here (usually, errors; normally, the field is empty)
Host: device IP address or hostname
Port: device HTTP/HTTPS port (from device settings, used to exchange service information, not data)
Channels
The channel monitoring section contains details about channel streams. Wherever relevant, there is information about both main and secondary video streams in separate columns.
Channels can have critical errors (red) and/or warnings (yellow). Warnings usually appear due to minor issues - e.g., missing substream, too big GOP length etc. You can ignore them by right-clicking item(s) and selecting Ignore warnings > (preference). If you want the warnings to be shown again, right-click the items and select Reset warnings. To reset all warnings, go to the Macula Console application settings (main menu > Settings) and select the corresponding option.
The Channels section has the following columns in the main area table (some are hidden by default):
Title: channel name
Server: the server currently handling the target channel's device
Device: the name of the underlying hardware piece for the target channel
Device model: underlying device type
Folder: if the channel belongs to an Folder, if will be displayed here
Status: current channel condition (e.g., Disabled means the channel has been turned OFF in the Channels section)
Status update time: timestamp of when the channel status was last checked
Online: shows channel uptime in percent compared to the total server run time
Video loss duration: shows if the video stream is not available; if yes, for how long the video stream has been lost
Kbit/s: stream bit rate (Kbits per second), separate columns for the main and secondary streams
FPS: stream frame rate (frames per second), separate columns for the main and secondary streams
Resolution: frame size in pixels, separate columns for the main and secondary streams
Codec: stream compression, for both main and secondary streams, separate columns for the main and secondary streams
GOP size: key frame interval, for both main and secondary streams (only if recording is activated), separate columns for the main and secondary streams
Recording: recording status ON/OFF (activated/not activated)
Recording error: if any
Frame cache: the amount of memory allocated for pre-recording of the target channel (total for all channel streams)
Stream storage: destination storage, for main, secondary, and edge streams
Information: additional information, if any, will be displayed here (usually, errors; normally, the field is empty)
Motion detection mode: currently used motion detection setting (disabled/camera-side/software-side high accuracy mode/software-side high performance mode)
IP: device IP address or hostname
Port: device HTTP/HTTPS port (from device settings, used to exchange service information, not data)
Jitter: frame latency; shows delay (in milliseconds) between expected and actual frame arrival time, separate columns for the main and secondary streams
Error: the latest error related to the channel (same as in Details, reset upon server restart)
Error date: the latest error timestamp
VA Enabled: video analytics status
VA Detector result: video analytics response (successful/error)
VA Device: device used for video analytics
VA Requested detection rate: detection frequency interval preset
VA Actual detection rate: actual detection rate
VA Average detection time: average time spent on detection
VA Average processing time: time spent on data processing
VA Processed frame count: how many frames were analyzed
VA Skipped frame count: how many frames were skipped
VA Processed frame rate: how many FPS attached video streams have
VA Skipped frame rate: how many FPS skipped
VA Detected frame rate: actual attached video stream FPS
VA Decoder result: video analytics decoder response
VA Decoder device: video analytics device (if any attached)
VA Codec: what codec is used to decode the attached stream
VA Average decoding time: how much time is spent on video stream decoding
For any channel, you can press the Notifications button on the upper panel to see the list of all channel-related errors, starting from last the server start. This list is purged upon server restart.
Note that Not activated recording status may both mean either that recording is not configured (disabled) or that recording has not been activated according to a schedule. The Activated status will appear if at least one of the channel streams is set to be recorded based on any configuration apart from No recording.
For any channel, you can also view the currently active recording profiles for all its st streams. To do this, press the Active recording profiles button in the upper panel. A dialog window will pop up, providing the information about the stream recording configuration. An empty list means no recording configuration is assigned to any of the channel's streams. This feature is useful for verifying if the recording schedule is executed correctly.
Streams
This section displays the properties of every recorded stream separately. For example, if a channel has both main and secondary streams with audio activated, there will be three entries. Replication streams are also displayed separately.
Channel: stream source, one of the existing enabled channels
Stream type: video, audio, motion, external data (from data sources), or VCA metadata
Status: current stream condition
Server: originating server
Device: originating device
Status time: last status update timestamp
Recording storage: target storage title
Recording directory: one of the target storages on the corresponding server
Last recording time: timestamp of the last recorded frame (useful for troubleshooting the recording)
Prerecording time: actual number of second held in the pre-recording buffer (may be lower then the setting if not enough memory, or 0 when not required)
Information: additional information, if any, will be displayed here (usually, errors; normally, the field is empty)
Note that the pre-recording time here may differ from your pre-recording setting in the recording configuration. Macula server applies smart logic here and traces situations when larger buffer is not necessary, or even preventing the system from normal operation. Thus:
if the currently used recording profile does not involve any pre-recording (e.g., Continuous recording, or event-driven with no defined E&A events), the buffer size will be reduced to 0
if there is not enough memory for all channels (the server is overloaded), the server will reduce pre-recording buffers; channels with largest frame cache size will have their buffers reduces first pf all
upon server startup, the buffer size is increased gradually for smoother start
Removed and disabled channels will not be shown under this section: only active recorder tasks are displayed here.
Archive Statistics
Here, you can see recording statistics for every channel (all streams and all storages included):
Channel: configured channel title
Begin date: the time of the very first recording for the target channel
End date: the time of the very last channel's recording
Duration: total archive duration for each channel
Data size: total archive size per channel, based on the time boundaries mentioned above
Data size per day: an estimate of how much space one day's recordings will take (average per day, calculated based on existing archive size)
Protected interval: total duration of the archive, which has been marked as protected, and will not be removed (erased)
Protected data size: total size of the protected archive
Server: the server where the data is recorded (convenient for filtered view based on the server)
Double-click any entry or use the View button on the top panel to display details for each channel: view the recording statistics per storage and with detailed information on every stream including audio, motion, VCA and external data feeds.
To review and unlock the footage that has been protected from erasing via Macula Monitor application, select the target channel(s) and click the Protected intervals button on the top panel.
From this dialog box, you can unlock the previously protected parts of the archive. If these recordings fall under quotas (storage or duration, server wide or individual), they will be erased immediately.
Storages
Statistics, properties and status for each storage unit are displayed here:
Title: storage label, either built-in or user-defined
Status: operation status of the target storage
Normal: the storage can be used for recording and there are no issues detected
Critical: the storage cannot be used for recording due to one or more serious errors, check the Information column for description
Server: storage location, i.e., on which server is configured
Status update time: last time the storage status was obtained (use the F5 button on your keyboard to refresh the list)
Path: storage path on its server of origin
Encrypted: indicates, if the target storage has encryption ON/OFF, or if there are encrypted storages without provided passwords (will be marked red)
Free space: how much free space in gigabytes is left on the disk
Free space %: same but shown as percentage in reference to the total storage size
Used space: the amount of disk taken by all data (including third-party data)
Last available time: the moment of time when the target storage was last available for recording
Total space: entire storage size
Information: additional details, if any (e.g., storage related errors)
Disk queue length: number of read/write requests (local storage only) *
*The Disk queue length column is useful for assessing the storage condition. It reflects either the current or the average queue length (whichever is higher) per disk. Generally, 5+ requests per disk indicate a bottleneck in the disk subsystem. 2+ requests per disk for a long period of time may also indicate bottlenecks. For RAID storages, the queue length is summarized, so, for example, for a system with 8 disks the queue length should not exceed 16. A value of 0 means the storage is not overloaded. An absent value (empty) means it cannot be retrieved: the field will be empty for network storages and for unreachable disks (e.g., server offline).
If one or more storages have issues, the corresponding section will have a red circle with the number of detected issues next to the section name. Inaccessible storage units will have no total and free space information.
User Sessions
This monitoring area displays currently active incoming Macula Monitor connections via both TCP and HTTP ports with the following details:
User: user account
Remote address: IP address and port of the client machine, from which the connection is made
Start time: session start time
Type: client application type, may be Macula Console, Macula Monitor, HTTP or Mobile
Software version: remote client version
a list of CPU, memory and network load stats from the remotely connected client machine, similarly to the ones in the Servers section
Disconnected sessions will automatically disappear from the list.
Video Walls
This section contains information about the currently connected video walls and their displays.
Data Sources
This section contains information about the configured data sources and their status:
Title: data source name
Status: current data source status
Server: the server that is receiving the data
Last data receive time: the timestamp of the last received bit of data from the target data source
Error: if any
Status time: status timestamp
Additional information: if any
External Services
If License Plate Recognition, Face Recognition and/or other external services (third-party integrations) are connected, their session properties will be displayed here: service name, remote address and involved user account.
Reports
This section contains health monitoring data for the configured automatic VCA and software counter reports.
If a report has been sent at least once (by schedule, not as a test), the last execution time is shown here. For the reports that are currently set to be emailed automatically, the next (scheduled) execution time is also displayed.
Access control
If any third-party services are connected, their connection properties and status will be displayed here.
OPC
If any third-party OPC servers are connected, their connection properties and status will be displayed here.
Indicators
All indicators configured in your system will be displayed here, so that you can see their status - all in one place.
GSM Modems
All GMS modems configured in the system will appear here:
Title: modem name
Status: current modem status
Server: the server that has the modem physically connected to it
Error: error code or text, if any
Status time: status timestamp
Additional information: error details, if any
MQTT Clients
All currently active MQTT clients will appear here. If the MQTT client is configured but is inactive, it will not be listed.
Databases
This section lists the time series databases associated with data sources and data channels.
Last updated