Data Sources and Data Channels
This part of the manual covers the feature of data sources in Macula. Data sources are a means of retrieving textual data from a serial source, such as a Point of Sale (POS)terminal. You can receive information in textual form from various third-party serial data providers like intercoms, bank machines, and a multitude of other devices capable of delivering data strings over the following connection types:
UDP port (remote connection)
TCP port (remote connection)
COM port (direct hardware connection to the server)
As a result, you get textual information, which you can store, analyze, react to, run queries to, and visualize in Macula Monitor.
The following entities exist in Macula Console for this feature configuration:
Data source: defines how the data are retrieved from the external source
Data source profile: describes how the data are parsed and extracted on the Macula side
Database: keeps the extracted data
Data channel: displays the data from the database and allows detailed DB search in Macula Monitor
Data channels require an additional license.
General Concept
Macula servers receive serial data from the defined sources, splits it into transactions and individual lines of text, extracts and stores keywords and values, combines the text with the video streams, and allows advanced search across the database.
Simplified configuration steps for the data reception from data sources can be briefly described as follows:
Create a data source, then
Create a data source profile, then
Link them together, and then
Assign the data source to the desired channel(s),
(optionally) Set up variables if you wish to react to certain keywords.
Make sure that the time precisely matches on both the server and client sides or the data source may not be displayed properly in the client application's live mode. If the system is disconnected from the internet, you may set a local NTP server or just adjust the time manually.
Gathered data are then stored and displayed embedded with the video stream from the channel(s) you choose to associate with it.
Starting from Macula version 1.23, there are additional entities available that allow building more complex scenarios: databases and data channels. These are not mandatory for receiving the serial data, yet they provide advanced means of storing and searching the data. Hence, you may find databases and data channels most useful when working with an abundance of complicated text like POS transactions, plus when there is a necessity to run complex textual queries on the data from the Macula Monitor side.
With databases, you can store your selected data (a part of the data stream) in a database with further refined search in the Macula Monitor application.
Data channels display text from data sources as a separate channel without underlying video, creating a convenient representation of the textual feed.
Macula Monitor also has a special playback mode featuring advanced database search.
Take the following steps to set up your advanced setup involving databases and data channels:
Create a data source (or multiple data sources), and
Add a data source profile,
Link them together,
Create a database,
Map variables from the data source profile to the DB, and
Add a data channel (or multiple) linked to the data source(s).
In this more complex scenario, you can still use variables to trigger events in the Event & Action scenarios.
Data Visualization
Data sources, data profiles, variables (mappings), and databases are configured in Macula Console. For users to access your data in Macula Monitor, you can:
Combine data with existing video channel(s) so that the text overlays the video in Macula Monitor. No additional entities are required, the text overlays existing video channel(s) and the combination is displayed in Macula Monitor in the same viewport.
Add data channel(s) to create a visualization for the transactions. Data channels will be displayed in a separate viewport but you can link them logically to video channels.
The first method is simpler and has usability restrictions. Data channels, on the other hand, provide additional functionality, but require a special license to operate. Both are described below in details.
Below, you will see how these entities are set up in Macula Console. For Macula Monitor part, please see the Macula Monitor user manual chapter on data sources.
Add Data Source
First, go to the Configuration section of Macula Console and choose the Data Sources component from the menu on the left. Then, click the + New data source button on the upper panel to open the data source creation dialog box.
Give your data source a comprehensive name. Then, choose the server that is going to be accepting the serial data. Leave the Data source profile field empty for now and select the desired transport for the data connection. You will need to return here and choose the profile after you create the profile itself.
There are three possible connection types: UDP port, TCP port and COM port.
TCP/UDP Connection
Choose this option if your data provider is configured to feed the textual data into a specific TCP or UDP port.
You simply need to specify the incoming port for the server to listen to in the settings here. Make sure the port is opened on all intermediate firewalls and not used by any other software on the server machine.
COM Port Connection
Choose this connection type if your data provider is connected directly into the Macula server's COM port.
For this configuration, you need to match the serial connection settings with those used from the device side. Normally, this can be checked via Windows Device Manager for your target COM port.
The following serial port settings should be specified:
COM port: serial port to read the data from, choose from the list of active ports
Baud rate: information transfer rate (9600 by default)
Data bits: the number of bits transmitted over the serial interface (8 by default)
Stop bits: the number of bits to specify the end of a byte; if you select data bits to be 6, 7, or 8, then the default value is 1 and the other available choice is 2; if you select data bits to be 5, then the only correct choice for the stop bits is 1.5
Parity: how the parity is checked (no parity by default)
Other serial port parameters are not required for the Macula server.
Click OK to save and close the data source creation dialog box. The next step is to create a data source profile to help the server determine how the textual data should be parsed.
Add Data Source Profile
On this step, you will load a portion of the serial text from the data source configured as shown above, and set up how the text will be parsed. That includes breaking the text into lines and transactions, and adding variable mapping.
In the Data Sources section, click the little arrow next to the data source creation button and choose the + New data source profile from the list. In the Details tab, enter a custom name for your data source profile and switch to the Configuration tab.
It is now necessary to use an example of the data strings to set up the profile. This part of the setup is the same regardless of what type of data source you have.
You can either use the actual data source you have configured on the previous step, or use a text file that contains the lines of data in the expected format (note that the same encoding should be used). Use the buttons on the right side of the text field to load the sample from either source. Use the Stop button to finish displaying new text from the data source (do this if you see that the loaded data lines are sufficient to facilitate the configuration process); use the Clear button to erase whatever is in the preview window before loading a different file or strings from a data source.
Based on the loaded sample, set the required parameters below:
Encoding: pick the correct encoding to ensure your text is readable
Line ending: choose what non-printable characters represent the end of the line, available options are:
CR - carriage return
LF - line feed
CR+LF - both together
Custom - user-defined symbol or group of symbols
If line delimiter in use is one of the standard non-printable ones (LF, CR or CR+LF) but you are not sure, whichever is used, press the Detect button for the software to identify automatically, which delimiter is present in the preloaded text. If you know that the provided serial data does not include any standard delimiters, choose the Custom option in the drop-down list and define your own delimiting character.
It is crucial that these settings exactly match the source settings. In case of a slight mismatch, your text may still appear readable but the lines may not be split correctly, causing errors in mappings.
Mappings
Mappings are pre-defined character combinations expected in the data flow. Typically, mappings are defined by regular expressions, which are used to extract required values of all sorts from the incoming text: identifiers, keywords, surnames, codes, etc.
Built-in Mappings
Macula server splits the incoming text into transactions. Each transaction has a certain structure, which can be visualized as follows:
-BEGIN TRANSACTION
//header starts here
<transaction fields are captured (e.g. POS ID, Cashier Name etc)>
-HEADER END
-BEGIN DETAIL
//item details
<detail1 fields are captured ( e.g. Item, Price, Quantity etc)>
-DETAIL END
-DETAIL BEGIN
//next item details
<detail2 fields are captured ( e.g. Item, Price, Quantity etc)>
-DETAIL END
-FOOTER BEGIN
//footer
<transaction fields are captured (e.g Total )>
//footer ends here
-END TRANSACTION
The CAPS text in the visualization defines the marks for the Macula parser that help it to split the text correctly. For each transaction and for each detail it is necessary to mark both beginnings and ends. For the transaction header, it is enough to mark the header end because the transaction beginning serves as the header start. For the footer, it is enough to mark the place the footer starts, as the footer automatically ends with the transaction. All of these marks are mandatory if you plan to use data channels and databases.
There are several built-in mappings (existing by default and non-removable) that help you define these BOUNDARIES:
BeginTransaction: specify text that designates the beginning of the data block (transaction)
EndTransaction: the very last data line of the expected data block
GetID: use a regular expression to determine and extract the data ID of each line (skip this if you only have a single data stream)
HeaderEnd: the text that marks the end of the header section, details are expected after it
FooterBegin: the text that designates the beginning of the footer section, and the end of recurring details
DetailBegin: how each detail (item) starts
DetailEnd: how each detail (item) ends
All of these mappings can be plain text or regular expressions, except for GetID, which must be a regular expression because it must capture variable values.
For the transaction start and end, you can specify regular expressions (regex rule) to define the start/end pattern instead of static text. You can also copy and paste the strings for transaction start and end directly from the sample preview field.
For the non-regex mappings, you can simply highlight the target text in the preview field, right-click it, and choose the mapping to bind the text to.
Custom Mappings
Additionally, you can create any number of mappings of the user variable type. These mappings use regular expressions to extract specific pieces of data into a variable, i.e. map variable text into pre-defined value containers. Custom variables have two usage scenarios:
variables allow you to extract the text and store it in the database. For this, you need to create the database with the desired fields, and then bind mappings to the DB fields.
E&A event Variable Condition. Variables can be used, for instance, to detect exceptions in the cash operation in case the data source is a Point-of-Sale system.
To create a new variable, click the + New button in the Mappings section, fill in the settings on the left and click Apply changes. The following settings are available:
Type: variable (cannot be changed)
Name: user-defined variable title
Text: a regular expression that defines what character combination should be extracted into the variable
Case sensitive: enable this if uppercase/lowercase is important (disabled by default)
Write to database: map the variable to a database field for storing and search (if you are not planning to use databases and data channels, ignore this option)
Load some text and then use the Test button to check if your settings work: if everything is fine with the setup, lines should be detected correctly, transaction start and end marked as such and variables extracted as specified.
Click OK to save and close the profile configuration dialog box.
Now, go back to your data source settings and specify the newly created data source profile that was missing in the initial configuration. You can use a single data source profile for different data sources. In this way, you link the two together, ensuring that the data received from the data source is passed to the data source profile for further analysis.
Video Overlay
You can put the serial text over the video so that they appear together in the Macula Monitor application. The text will be also available for simple search in the special tab, Data Sources, in Macula Monitor.
Under Configuration, go to the Channels section and pick a channel that needs to be associated with a data source. Double-click the channel to open it for editing and switch to the Video overlays tab. Click the Configure video overlays buttons to bring up an additional dialog box.
Here, different kinds of video overlays can be configured. The Data overlay element is already present by default - it is the pink frame overlaying your video.
On the right side, you will have the available settings. Click the Change button and select one of your pre-created data source for this channel. If the serial data are currently arriving, you will see the text appear in the frame. Then, adjust the settings on the right to define where and how the text should appear on top of the video stream.
Use the visual control - the pink rectangle - to specify the text output area within the video stream viewport. There are other types of video overlay elements available in the same window, but the one you need is called Data overlay and it exists by default, so you do not have to add it. There are also numerical coordinates on the right side of the preview for your reference: you do not have to edit them manually.
Also, note that the visual control cannot be removed from the configuration; if there are no data to display, this element simply will not exist in the Macula Monitor application. The visual control is only present in this configuration window for your convenience. There is exactly one Data overlay element per viewport, meaning that each channel can only have text output from a single data source.
Make sure the video from the target channel is available before setting up the text overlay in order to ensure correct text placement.
Additional settings:
Line count: set the desired number of text lines to be fitted into the overlay area (default value is 20); this will affect the text font size
Timeout: the amount of time in seconds for every line of data to stay on the screen at maximum (older text will disappear and text will be shifted upwards)
Text color: choose a color for the overlay text from the standard palettes (default is white)
Data ID: data source identifier, if present (leave empty if not used)
In the Data ID field, you can enter the source identifier in case the serial data are being split between two or more channels; leave the field blank if this channel is the only destination for all the data received from the selected data source.
When ready, click OK to save and close the data overlay dialog box, then click OK to save and close the channel configuration window.
Now your textual data from the data provider will be displayed overlaying the video stream in Macula Monitor and both live view and in the instant/regular archive playback mode in case the corresponding option is enabled in the recording profile for the target channel; also, it will be possible to search through the recorded data.
The overlay text will only be visible in live view and regular/instant playback, and will not be displayed in other views (e.g., smart search). In order to export video from the archive with the text overlay, use MKV file format + VP8 compression + hard or soft subtitles.
If you do not wish to combine the serial data with video, check an alternative method described below. It uses databases to store the serial data instead of the video archive, and provides a dedicated visualization in the form of a data channel.
Time Series Databases
Data received from the data sources can be stored in the databases of a proprietary type, which use the time series database principle. Time series databases are optimized for simple write and search operations, effectively holding a large quantity of the serial data that arrive over time (and are usually written in a series and not injected somewhere in the middle).
Each database (DB):
is bound to one or more data sources,
receives data from these data sources as defined by mappings in the data source profile,
writes the data locally (on the same server with the data source), and
performs search based on user queries from Macula Monitor and returns the requested information.
Create New Database
To add a new database via Macula Console, make sure you are in the Configuration section > choose Data Sources on the left > click the down arrow next to the + New.. button and choose the New time series database option.
In the new DB dialog box, fill in the database settings.
In the Details tab:
Title: user-defined database name that will appear in Macula Monitor
Enable: if marked, the database will be available for write & read
Maximum number of days: entries older than the specified number of days will be removed
Writable interval: the maximum age of a new entry, older entries will be discarded
The data are normally written to the database real-time. However, if there are delays in the synchronization due to network availability issues or other reasons, you may want to write older entries. For this purpose, each database has a limited writable interval, during which it will accept new entries with non-real-time timestamps. Keep this interval shorter if you prefer a faster DB, and set a longer interval if you know that some of the data may arrive with a significant delay (e.g., from remote locations). Default value: 1 day, max value: 99 days.
In the Membership tab, you can put the database into one of your visual groups for a more convenient presentation in Macula Monitor.
After you have set the general settings, switch to the Database tab. Here, you need to create the database structure using the following guidelines:
each database can have 2 (two) tables
the first table stores so called transactions (larger chunks of information the serial data are broken into)
the second table holds transaction items (repetitive individual elements of the transaction)
each of the two tables can have up to 32 fields, for example:
transaction fields (main table): POS ID, door ID, transaction ID, operator's name, total, etc.
transaction item fields (details table): product ID, product name, amount, subtotal, etc.
fields that have repetitive values from a fixed list (e.g., operator's name or item name) can have reference tables. For such fields, their values are replaced by keys in the main table, while longer textual values are kept in the reference table, speeding up the DB operation.
When you create a new DB, it offers you to create the main table first:
Title: user-defined table name that will appear in Macula Monitor. The title may contain a wide range of characters: non-Latin alphabets (e.g., Cyrillic letters), white spaces, and special characters.
Alias name: internal table name that may only contain Latin letters [A-Za-z] and numbers [0-9].
After changing the settings, click Apply changes: your table name will appear in the DB structure on the right.
After that, use the down arrow net to the + New field button on the right to add the second table (details table) using the same logic. After you have done that, or if you only plan to have one table, use the + New field button to add fields to both tables:
Title: user-defined field name that will appear in Macula Monitor. The title may contain a wide range of characters: non-Latin alphabets (e.g., Cyrillic letters), white spaces, and special characters.
Alias name: internal field name that may only contain Latin letters [A-Za-z] and numbers [0-9].
Data type: choose if the value will be stored as text/integer/long/double. Textual fields can have reference tables.
Create reference table: for text fields only; mark this option if you know that the data field value belongs to a finite list of values (e.g., operator's name). Do not use this option if the field value is [mostly] unique.
Note that you do not have to describe everything you expect to receive from the data source, but rather just the values you want to store and search in Macula Monitor.
After having created all the tables and all the fields, click OK to save and close the dialog window. The newly created time series database will appear in the list in the Data Sources section. Note that the database will only appear in the Monitoring section of Macula Console and in Macula Monitor after there are some actual data written to it. For this, you need to create at least one data channel and link it to the same data source.
Data Channels
Data channels are a way to visualize the serial data received from data sources without combining it with the video in one viewport. Thus, you get your data displayed neatly in a separate viewport next to the video channels.
Create Data Channels
To create a new data channel in Macula Console, go to the Data Channels section under Configuration. On the top panel, click the + New Data Channel button
In the Details tab, fill in the following settings:
Title: user-defined data channel name that will appear in Macula Monitor
Data source: the source of the text to be displayed
Data ID: the actual identifier of the data stream extracted by the mapping GetID (defined in the data source profile)
Main stream recording configuration: how the data will be recorded (main stream only)
Main stream storage: destination storage
Not that it is mandatory to fill the Data ID field if your data source has the GetID mapping defined. When the data source parser encounters a GetID variable value, it searches across all data channels for the target data channel and writes the corresponding data to the database. If the GetID mapping is defined but there is a data channel with an empty Data ID field, this data channel will not have any data fed to it.
In the Members tab, you can pair the data channel with two types of resources:
channels: one or more video channels that will pop up together with the data channel when you search across the database and get results from this data channel
user buttons: paired user buttons will appear in the data channel viewport, next to its name, so that you can click them to trigger the related event
By making video channels members of the data channel, you facilitate the textual search in Macula Monitor. For example, if your data channel is a POS terminal, it would make sense to pair data channels to video channels that face the cashier's desk.
In the Membership tab, similarly to the regular channels, you can put your data channels into groups (for permission handling) and visual groups (for grouping on the Macula Monitor side).
Finally, in the Permissions tab, just as for any other resource, you can define access permissions for your users and user groups.
Data channels are similar to video channels; hence the permission list is very similar:
View live video: permissions related to the live streaming (Live tab in the server configuration).
In the small window that pops up, choose the desired items, then click OK to proceed: the items will appear in the Data sources section and you will be able to edit them as usual.
Note that you will need to create data sources and data channels to complete the setup.
Last updated