User Asset Checking
The purpose of this feature is to limit the visibility of a user on a set of hosts present on SGBox, showing only those that are part of an asset to which his user is assigned.
Therefore at the time we need to show a user of SGBox only certain hosts, we proceed by going under the assets section thus creating a new one where we group, sources, modules, and users. Once that is done we go back to SCM > advanced options > user asset checking and activate the flag in question.
To conclude: the user will only be able to see the list of hosts that are part of the asset.
In order to guarantee the correct health of SGBox system, you can set different alerts:
Requirements:
- A mail server must be configured: Configure a mail server
Connect at the SGBox web interface on SCM > Advanced Options
- Disk Full: send an alert if the threshold is exceeded
- Load Average: send an alert if the load average is greater than the specified value
- Collector log file: send and alert if the collector queue has more items than the specified value
- Agent Inactive: send and alert if the agent do not contact SGBox for more than the the specified minutes
Here the details and the message received for every options:
Disk Full
Here the email you can receive if the threshold is exceeded:
Your SGBox storage utilization is higher than the configured threshold (current: 94% / threshold: 85%) This message will be sent every hour. To free up storage you can change the retention of the online, event or RAW logs to remove old logs. You can of course resize your HDD using the CLI interface by logging in with the 'cli' user to reserve more space to your logs. If you think that 85% is not a correct threshold you can modify this value in SCM > Advanced options > Disk full at X% to avoid this message. Don't forget that if you raise too much the threshold the disk could quickly become full and you won't be able to collect logs anymore, potentially damaging SGBox. Please refer to this link http://www.sgbox.it/knowledge-base/extend-full-disk/ for more information on resizing the SGBox HDD. Refer to this link http://www.sgbox.it/knowledge-base/sgbox-data-retention/ for more information on data retention. Warning: filling up your HDD could lead to unexpected SGBox behavior or damage your SGBox installation. Avoid this situation. Warning: assigning a too short retention to your RAW logs could lead you to violate your country regulations, if you are using SGBox to manage your logs in compliance with them. Change the retention period of the RAW logs with care. Please perform a backup of your data before changing the retention period.You can use one of the SGBox backup applications to perform backups (SCM > Applications > Backup) Note on HDD resize: depending on your virtualization platform, SGBox may need a restart to detect the new HDD size.
Load Average
Here the email you can receive if the threshold is exceeded:
Your SGBox CPU and I/O utilization (load average) is higher than the configured threshold (current: 3.46 / threshold: 3) This message will be sent every hour. This could be a normal behaviour caused by an high log collection rate. If you see this message more than 3 consecutive times or randomly during the day, please consider to increase the hardware resources of your SGBox instance. If you think that 3.46 is a correct load average, you can modify the threshold value in SCM > Advanced options > Load Average > X to avoid this message<
Not Received Logs
Here the email you can receive if the threshold is exceeded:
This email will be sent every 5 minutes for all the hosts. If you want to set a custom value for a specific host or stop the alert after a while you can go on: SCM > Network > Hosts. Select the involved host and click on “Alert Log“.
Specify the “Minimum time” that could be different from the default and the “Maximum time“. After the Max time no alert will be sent for the specified host.
Collector Logs
Here the email you can receive if the threshold is exceeded:
Collector main queue contains 900 items This message was sent because the SGBox main queue is collecting data that is ordered too slowly. This could be a temporary problem, but if this message occurs frequently, we recommend to check if your hardware is adequate for the amount of logs you are sending to SGBox.
You can check it also by using the SGTop
You can set and higher value or check the troubleshooting section in order to solve the problem.
Engine Logs
Here the email you can receive the the load average threshold is exceeded:
This message was automatically generated by SGBox because a log messages queue has become busy. LOG_input_default queue contains 20 items This problem does not require immediate action. Just check if log flow looks normal by connecting to the SGBox web interface and navigate to LM->Analysis->Class/pattern analysis and erify if your events look consistent. If you receive this message too often, please check if your hardware is adequate for the amount of logs you are sending to SGBox.
Agents Inactive
In order to receive the notification the agent must be selected in: SCM > Network > Host in the tab “Agent Status“.
Here the email you can receive if the threshold is exceeded:
The following agents have been inactive for at least 12 minutes Last connection Delta Last sent log Delta Host Hostname Network Network description ------------------------------------------------------------------------------------------------------------------------- 2020-10-27 13:30:39 898730m 2020-10-27 13:30:33 898730m WIN7 WIN7 Host LM Host rilevati da LM 2021-06-10 15:45:48 573155m 2021-06-10 15:45:56 573155m LAB2-WIN LAB2-WIN Host LM Host rilevati da LM]]>
The SGBox Data Retention
In this section we will explain how SGBox stores logs.
The logs received by SGBox are called “RAW logs”. The raw logs represent exactly what the data sources send to SGBox.
When the raw logs are received, they’re stored in the SGBox storage system, the “Online storage”. You can access and make searches on the Online raw logs in LM > Analysis > Historical Search
The logs are then analyzed in order to match them with known events. If SGBox founds a match, this information is saved in another section of the internal storage. You can access the matched events in different ways: the easiest is LM > Analysis > Class/Pattern Analysis.
The raw data is also “sealed” in text files that are taken from the SGBox storage and kept in a compressed, encrypted and signed format for security and regulatory compliance reasons. The encrypted data is also associated to a timestamp. The encrypted data cannot be directly accessed by the user, it can only be exported with the backup application.
You can configure different retentions for each storage in SCM > Advanced Options.
You can set the Encrypted raw logs retention in the following section
You can set the Online raw logs retention in the following section.
You can set the Events retention in the following section.
SGBox Log Management and Retention
This section explain how SGBox manage logs and difference between: online logs, encrypted logs and events.
Requirements:
- SGBox version 5.0.4 is required.
From the beginning: SGBox receives log data from different inputs / data sources and by using different protocols.
+Online Logs
The raw data from each data source is associated with a unique tag and placed in a local storage to SGBox. In this position, which we will call the “online log repository”, the data is kept in a compressed and indexed format, so that the user can consult it at any time through the Historical search ( SCM -> Analysis -> Historical search ).
Online data cannot be modified in any way by the user and at regular intervals data is extracted, divided by data source and placed in another storage, not accessible to the user, in encrypted format.
+Encrypted Logs
Data present in the “encrypted log data storage” are encrypted using GPG standard and cannot be modified in any way by the user. Each encrypted block is also compressed, signed and associated with a timestamp linked to the encryption key.
User is always able to verity the data integrity of the encrypted data by using a dedicated SGBox view ( LM -> Configuration -> Encryption ).
The only operations that the user can perform on this data are
1. define a retention of the data that can be different for each data source ( SCM -> Advanced Options or LM -> Network -> Hosts ).
2. transfer a copy of the data to online storage ( LM -> Configuration -> Online logs manager SGBox vers. > 5.0.4 )
3. backup the data using applications (SCM -> Applications ).
+Events
While data is collected, it is also analyzed by a series of parsers capable of extracting meaningful information from raw data, transforming it into what SGBox calls “events”.
Events, just like online logs, are freely accessible by the user. Events generated by log parsing, report the same unique id the logs were tagged with, to definitively bind the event and the log that generated it. User can rebuild events, enriching them with new information, build correlation rules and dashboards on them, while the algorithms – always active on SGBox – analyze events searching for anomalies in behaviors and volumes.
This figure shows the log storage architecture and the access rules to the online log and events data (Remember: only online logs and events can be accessed by user)
Note that data exported to the external storage always leaves SGBox in encrypted and compressed form.
When restoring data from the external storage you can always check for data consistency by validating the signature for each block of data.
You can set the retention from SCM > Advanced Options
In the Retention section
Configure a mail server
To configure the mail server you need to connect to the web interface of SGBox: SGBox > SCM > Advanced Options
Go down below to the Email configuration section.
Once configured, click the Save button. It is possible to verify that all settings are correct by clicking on Send a test email.
In the Reveiver address, multiple address can be added, separated by “,” (comma). An example:test@domain.com,exmple@mydomain.org