Some things are simply often asked. Here we provide the answers!
If you have questions that should be explained here, please let us know.
Issues, problems, enhancements and feature requests are managed with our project workflow. For extended communication visit ELMA discussion group.
For syslog specific questions please also have a look at rsyslog and syslog-ng FAQs. Further informations and tips arround log management you can find in ElasticSearch tutorials.
ELMA user and admin interfaces should work in all modern web browsers as long as they support SVG.
You may need to clear your browsers cache after upgrading to see all of the new features.
If you are running into ssl browser errors you may need to delete your ELMA certificate inside your browsers server certificate store.
A busy system will blow up ElasticSearch or rsyslog working space with data relatively quickly.
It is strongly recommended to separate or to extend these LVM Volumes with a sizing depending on your specific requirements using integrated Logical Volume Manager (LVM) inside ELMA CLI.
A good procedure for File System Sizing immediately after deployment:
Stop services
service rsyslog stop
service elasticsearch stop
Rename ElasticSearch data directory from /var/lib/elasticsearch to /var/lib/elasticsearch-old
mv /var/lib/elasticsearch /var/lib/elasticsearch-old
SUSE does not auto mount non-root LVM volume groups on boot, so be sure to run
chkconfig boot.lvm on
Add another virtual disk, reasonably sized (I make it 100GB to start). Create a new volume group, and a new logical volume (using file system type EXT3 or better EXT4) and mount it at /var/lib/elasticsearch
Move the files from /var/lib/elasticsearch-old to the new /var/lib/elasticsearch
cp -Rp /var/lib/elasticsearch-old/* /var/lib/elasticsearch
Set the owners and groupname back, just in case
chown -R elasticsearch:root /var/lib/elasticsearch
chgrp -R elasticsearch:root /var/lib/elasticsearch
Check path to ElasticSearch data directory
grep "ES_DATA_DIR=" /etc/elma/elma.conf
ES_DATA_DIR="/var/lib/elasticsearch"
System reboot
reboot
Final changes should look similar to this:
Changes to partitioning:
* Create volume /dev/sda3 (786 GB)
* Extend volume group systemVG with /dev/sda4
* Extend logical volume /dev/systemVG/LVRoot to 759.80.79 GB
* Extend logical volume /dev/systemVG/LVopt_ to 10.00 GB
* Extend logical volume /dev/systemVG/LVvar_log_ to 29.98 GB
* Extend logical volume /dev/systemVG/LVvar_spool_ to 10.00 GB
ELMA search frontend get the logging events from ElasticSearch database which stores the indexed data per default in /var/lib/elasticsearch.
Watch how much your logs grow in a day. For example, if /var/lib/elasticsearch grows from 10 GB to 15 GB in a day, this is 5 GB/day.
df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 32K 1.9G 1% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 3.0M 1.9G 1% /run
/dev/mapper/systemVG-LVRoot 60G 42G 17G 72% /
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 3.0M 1.9G 1% /var/lock
tmpfs 1.9G 3.0M 1.9G 1% /var/run
/dev/sda1 183M 35M 139M 20% /boot
/dev/mapper/systemVG-LVvar_log_ 30G 3.2G 25G 12% /var/log
/dev/mapper/systemVG-LVvar_spool_ 29G 16G 13G 55% /var/spool
Divide your available storage by the daily usage to determine the number of days of logs you can keep. For example, if you have 100 GB of available storage and you are using 5 GB/day, you can keep 20 days of logs.
100 / 5 = 20
Edit “/etc/elma/elma.conf”
######### Set the number of days ElasticSearch indexes should be hold #########
ESARCHIVEPERIOD="20"
Reconfigure ELMA to use new setting:
/usr/share/elma/bin/elma-reconfigure.sh --all
For all patching und upgrading tasks the usage of ELMA CLI is more secure then the usage of ELMA webgui because of possible apache and php reloads during patches and upgrades of underlying operation system
/usr/share/elma/bin/elma-reconfigure.sh --patch --update
Interactive update from ELMA CLI
zypper --no-gpg-checks --gpg-auto-import-keys patch
System reboot
reboot
There are to much changes from openSUSE 12.2 to 13.1 especially with openSUSE startup scripts and logging features. It’s strongly recommended to upgrade all of your ELMA appliances (see below).
There will be no further support for openSUSE 12.x appliances.
Upgrade howtos can be found on openSUSE support site.
There will be no further support for openSUSE 12.x appliances. Please upgrade to openSUSE 13.1!
Edit your syslogd.conf file, usually found in /opt/syslog-ng/etc/syslogd-ng.conf and make sure you have a source called s_all:
source s_all {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" program_override("kernel: "));
};
Next configure the protocol and port, and put them in a destination entry, being sure to specify TCP. Replace the “192.168.0.1” with the address of your ELMA server.
destination d_elma {
tcp("192.168.0.1" port(514));
};
Next tell syslog-ng to forward syslogs from “s_all” source to the “d_elma” destination. Note: This log statement must be before any “final” log statements, so make this the first log statement.
log {
source(s_all); destination(d_elma);
};
Your configuration should now look as follows:
source s_all {
internal();
unix-stream("/dev/log");
file("/proc/kmsg" program_override("kernel: "));
};
destination d_elma {
tcp("192.168.0.1" port(514));
};
log {
source(s_all); destination(d_elma);
};
Now, restart syslog-ng:
/etc/init.d/syslog-ng restart
Test sending events to ELMA by using the command line tool logger:
logger "my little pony"
Log into the ELMA search interface. Search for your events, starting with “my little pony”.
On sender site:
# ######### Event forwarding to ELMA rsyslogd server ##########
#
# plain TCP event forwarding
#
*.* @@192.168.0.1:514
On receiver site: nothing to do
On sender site:
# ######### Compressed event forwarding to ELMA rsyslog server ##########
#
# plain TCP event forwarding with experimental framing
# maximum zlib compression rate (z1 - z9)
#
*.* @@(o,z9)192.168.0.1:514
On receiver site: nothing to do
By default rsyslogd makes reverse DNS requests to resolve the hostname and stores it instead of sender IP. The is no further customization on ELMA appliance necessary.
You have to make sure that your ELMA box has right local DNS server settings!
You could test reverse DNS resolving using dig on ELMA linux shell:
dig @localhost -x 204.152.184.167 +short
One of the quickest commands to verify that ntpd is still up and running as desired is:
ntpq -p
To compare time offsets between local system time and time taken from your ntp server:
ntpdate -d server
To synchronize your system time manually:
systemctl stop ntp.service
ntpdate server
systemctl start ntp.service
Reading the clock:
hwclock --show ; date ; date -u
The commands above show, in succession:
rsyslog and Kibana time stamped indices use your selected time range to create a list of indices that match a specified timestamp pattern.
Please also note that rsyslog indices should rollover at midnight UTC. Therefore it is strongly recommended to set system clock to UTC.
UTC (Coordinated Universal Time) is the international time standard. It is the current term for what was commonly referred to as Greenwich Meridian Time (GMT). Zero hours UTC is midnight in Greenwich, England, which lies on the zero longitudinal meridian. Coordinated Universal Time is based on a 24 hour clock; therefore, afternoon hours such as 4 pm UTC are expressed as 16:00 UTC.
It is strongly recommend that the system clock (as well as the CMOS/BIOS clock) is set to UTC/GMT.
System time and date:
timedatectl status
This way Linux can keep the user space clock on the correct time when the change for Daylight Saving Time occurs. The safest way is to set the CMOS clock to UTC before beginning the installation process.
Rsyslog timestamp formatting:
date-rfc3339 2010-12-05T02:21:41.889482+01:00
date-rfc3164 Dec 5 02:21:13
date-pgsql 2010-12-05 02:27:34
date-mysql 20101205022845
In some case if you are using DHCP (dynamic or static) there is no /etc/host entry for your ELMA host. You have to insert your hostname with correspondig IP address.
Better you change your network configuration to static for server environment. Make sure you have also correct DNS resolving (especially reverse resolving) for your appliance.