Published: Thu 24 August 2023
Updated: Fri 19 April 2024
By Don Cohoon
In linux .
tags: systems
Watch Notification System - v 2023.09.5
Finally got my alerts working, so when a host goes haywire I get an alert on the Phone and Cloud, with E-Mail as backup.
alert.sh and log.sh are meant to run a few times per hour
and send alerts if watch thresholds are exceeded.
alert.sh - Usually runs on one host and monitors other hosts, using a normal user ssh tunnel.
log.sh - Usually runs on each host as root.
When any new file is uploaded into the
new alert user space, an entry will be written
to the Conversation with the name and link
to that file.
On your Phone you can install NextCloud Talk, log in
as the new alert user and recieve notifications.
Bonus: Your watch will alert you too!
On your Phone you can install NextCloud Sync, log in
as the new alert user and read notification files.
Here is the file structure:
/ home / bob / watch
.
├── alert
│ ├── vm5 . util . 20230822112005. uploaded
│ └── vm7 . util . 20230822130010. uploaded
├── alert . sh
├── db
│ ├── db - 18.1 . 40
│ ├── oracle_berkely_DB - V997917 - 01. zip
│ └── readme
├── deploy . sh
├── df . sh
├── geturl . pl
├── log
│ ├── cloud . log . 0
│ ├── cloud . log . 1. gz
├── log . sh
├── mail . sh
├── nbs
│ ├── db . c
│ ├── db . o
│ ├── INSTALL
│ ├── Makefile
│ └── ...
├── nbs . tar
├── readme
├── retail
│ ├── Makefile
│ ├── retail
│ ├── retail . c
│ └── ...
├── retail . tar
├── savelog . sh
├── status
│ ├── apache - error
│ ├── apache - error . db . cnt
│ ├── apache - error . db . idx
│ ├── apache - error . db . rec
│ ├── apache - error . db . upd
│ ├── apache - error_new . txt
├── sync . sh
├── util . sh
└── watch . sh
/ home / bob /. config
├── watch
│ ├── hosts . txt
│ ├── config . txt
│ └── df . vm7
Installation
https://github.com/dfcsoftware/watch
Copy all files to ~/watch, or whatever directory you like,
just change this documents' references of
/home/bob
to
your directory.
Delete any lines in files /etc/issue and /etc/issue.net as they will interfere with function monitors doing ssh into a host causing alerts every time.
Files: /etc/issue, /etc/issue.net
$ sudo -i
$ > /etc/issue
$ > /etc/issue.net
Configuration
Create config directory structure:
$ mkdir -p ~/.config/watch
Create config file:
File: ~/.config/watch/config.txt
export CLOUD_USER =< nextcloud user >
export CLOUD_PASS = "<nextcloud password>
export CLOUD_DIR = alert
export CLOUD_SERVER = "https://www.example.com/nextcloud"
export CLOUD_LOG =/ home / bob / watch / log / cloud . log
export LOCAL_DIR =/ home / bob / watch
export SSH_USER = bob
export LD_LIBRARY_PATH =/ usr / local / BerkeleyDB . 18.1 / lib : $ LD_LIBRARY_PATH
Make sure the LOCAL_DIR/alert exists
$ mkdir -p ${ LOCAL_DIR } /alert
Make sure CLOUD_LOG is writeable
Create an hosts.txt list of hosts to monitor
File: ~/.config/watch/hosts.txt
# Host ssh Remote Remote
# Port Script Home
# ------ ------ -------- -----------------------
vm1 223 0 /home/bob
vm2 224 1 /home/data/bob
#
# Remote Script: 1=run moniter script that is on remote machine
# 0=run monitor script on local, through ssh tunnel
Schedule alert.sh in cron
i.e.: every 20 minutes
File: /etc/cron.d/alert
# Run the alert analysis
SHELL =/ bin / bash
PATH =/ sbin : / bin : / usr / sbin : / usr / bin
MAILTO = "bob@bob.com"
*/ 20 * * * * root / home / bob / watch / alert . sh
Copy ssh keys to remote
If remote monitoring is desired;
Generate and copy the linux ssh keys.
$ ssh-key-gen
$ ssh-copy-id <remote hosts>
Alert Functions
This is a seperate file for each functional alert, and sometimes host.
Remote Hosts need their own config file(s)
$ mkdir -p ~/.config/watch
Example of df functional monitor. Each script will describe it's own.
File: ~/.config/monitor/df.
Usage-Percent-Limit File-System Mount-Point
34 "/dev/mmcblk1p2" "/"
38 "/dev/mmcblk1p1" "/boot/firmware"
16 "/dev/md0" "/mnt/raid1"
Dependencies:
dnf=RedHat; apt=Debian
$ sudo dnf install pcp cockpit-pcp python3-pcp # default from cockpit web install
$ sudo apt install pcp cockpit-pcp python3-pcp # default from cockpit web install
# - Systemd Services:
# pmcd.service
# pmlogger.service
# pmie.service
# pmproxy.service
#
$ sudo dnf install pcp-export-pcp2json # pcp2json
$ sudo apt install pcp-export-pcp2json # pcp2json
# Debian 11 may need to add;
# File: /etc/apt/sources.list.d/unstable.list
# deb http://deb.debian.org/debian/ unstable main
# Also may need to run: pip install requests
$ sudo dnf install pcp-system-tools # pmrep
#
$ sudo dnf install jq # json parser
$ sudo apt install jq # json parser
#
$ sudo dnf install jc # json commands
$ sudo apt install jc # json commands
#
$ sudo dnf install ncdu # Text-based disk usage viewer
$ sudo apt install ncdu # Text-based disk usage viewer
#
$ sudo dnf install nmon # Text-based system utilization viewer
$ sudo apt install nmon # Text-based system utilization viewer
Default is to send an e-mail if limits are exceeded
To stop E-Mail, add a SEND_MAIL export to the config.txt file.
File: ~/.config/watch/config.txt
NextCloud Flow Notifications
Reference:
savelog.sh
This is used to save off several copies of the last sync.sh process sending alerts to the NextCloud server.
The logs are better viewed using the lnav Linu package. lnav ~/watch/log/
This is released on many OS packages, but not all,
so it is included here. Thanks very much to the original authors!
Reference:
log.sh - Log Watcher
This script runs as root on each node to search for Never Before Seen (NBS) entries in a log file.
It needs to be scheduled in cron.
The flow is:
cron runs log.sh
File: /etc/cron.d/logwatcher
# Log - Watcher
PATH =/ usr / lib / sysstat : / usr / sbin : / usr / sbin : / usr / bin : / sbin : / bin
MAILTO = "bob@bob.com"
# Run Log watcher
*/ 20 * * * * bob / home / bob / watch / log . sh
log.sh reads config file ~/.config/watch/logwatch.\<hostname>
File: logwatch.vm7
# File: logwatch.vm7
# DB File Alert Email Filter
# ------------------------- ----------------------------------------- ------ ------ -----------------------------------#
apache - access / var / log / apache2 / access . log N Y geturl . pl skip_local_ips . sh
apache - error / var / log / apache2 / error . log Y Y geturl . pl
apache - other_vhosts_access / var / log / apache2 / other_vhosts_access . log Y Y geturl . pl
New lines in File are read by retail, filtered by any and all Filters supplied.
DB is checked if this has been seen before and can be ignored.
Any remainging lines are sent to the alert directory and an Email sent, if Y in config.txt.
A sync for new alert files is done, sending new files to the cloud (NextCloud).
NextCloud will send a Talk alert to the CLOUD_USER for new files. These are viewable on a Phone or Watch, if running the Talk app.
The following software packages have to be installed and compiled locally:
Refer to the log.sh script for instructions.
Reference:
Trouble Resolution
armv7l Debian 10
The pmlogger systemd daemon had issues being installed, and the following log instructions solved it.
Aug 08 11 :07 :39 bob .example .com systemd [1 ]: Starting LSB : Control pmlogger ( the performance metrics logger for PCP ) ...
Aug 08 11 :07 :43 bob .example .com pmlogger [913 ]: / etc / init .d / pmlogger : Warning : Performance Co - Pilot archive logger ( s ) not permanently enabled .
Aug 08 11 :07 :43 bob .example .com pmlogger [913 ]: To enable pmlogger , run the following as root :
Aug 08 11 :07 :44 bob .example .com pmlogger [913 ]: update - rc .d - f pmlogger remove
Aug 08 11 :07 :44 bob .example .com pmlogger [913 ]: update - rc .d pmlogger defaults 94 06