Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix inconsistency between BACKUP_ITEMS and USE_ #115

Closed
tonyskapunk opened this issue Aug 25, 2017 · 3 comments
Closed

Fix inconsistency between BACKUP_ITEMS and USE_ #115

tonyskapunk opened this issue Aug 25, 2017 · 3 comments
Labels
bug
Milestone
2.0.0

Comments

@tonyskapunk
Copy link
Contributor

Back in issue #94 when working on the documentation and configuration of the settings, it was clear that there is a inconsistency between the list of BACKUP_ITEMS and the USE_ variables (#94 (comment)), it's required to be more consistent on those settings, likely to deprecate the use of BACKUP_ITEMS or create it in a dynamic way based on the USE_ variables.
@tonyskapunk tonyskapunk added this to the 1.2.0 milestone Aug 25, 2017
@tonyskapunk tonyskapunk modified the milestones: 1.2.0, 1.3.0 Oct 6, 2017
@tonyskapunk tonyskapunk modified the milestones: 1.3.0, 1.4.0 Nov 14, 2017
@tonyskapunk tonyskapunk mentioned this issue Dec 20, 2017
Closed
@tonyskapunk tonyskapunk mentioned this issue May 10, 2018
Closed
@tonyskapunk
Copy link
Contributor Author

I'm thinking in deprecating BACKUP_ITEMS. recap and recaplog are the two scripts that make use of it.

  • recap - Uses this setting for the backupfunctionality, that is, copy the last set of logs from BASEDIR into BASEDIR/backup. This can be generated dynamically regardless of the settings in /etc/recap.conf

  • recaplog - Uses this setting for compressing/moving logs daily. This also can be generated dynamically regardless of the settings in /etc/recap.conf.

The implementation of this will happen as part of #155 where recap would allow plugins/modules to be written.

@tonyskapunk
Copy link
Contributor Author

I have implemented the deprecation of BACKUP_ITEMS in branch tonyskapunk/issue155

Being able to dynamically backup/archive/compress/email is needed in order to add more reports through plugins/modules. That's the reason why it was implemented in there.

Documenting in here the tests performed:

Testing Backup:

2018-05-18 11:36:44-05:00 [INFO] --- Starting recap[15710] ---
2018-05-18 11:36:44-05:00 [INFO] -- bash info: 4 4 19 1 release x86_64-unknown-linux-gnu
2018-05-18 11:36:44-05:00 [INFO] recap[15710]: Created lock file: /var/lock/recap.lock
2018-05-18 11:36:44-05:00 [INFO] Starting check for disk space
2018-05-18 11:36:44-05:00 [INFO] Ended check for disk space
2018-05-18 11:36:44-05:00 [INFO] -- Taking backup, storing reports in /var/log/recap/backups
2018-05-18 11:36:44-05:00 [INFO] Starting backup of reports
2018-05-18 11:36:44-05:00 [INFO] Last run was on: 20180518-113616
2018-05-18 11:36:44-05:00 [INFO] Reports found: [ fdisk mysql netstat ps pstree resources ]
2018-05-18 11:36:44-05:00 [INFO] Ended backup of reports
2018-05-18 11:36:44-05:00 [INFO] recap[15710]: Caught signal - deleting /var/lock/recap.lock
2018-05-18 11:36:44-05:00 [INFO] Execution time: 0s
2018-05-18 11:36:44-05:00 [INFO] --- Ending recap[15710] ---

See it backed up fdisk, mysql, netstat, ps, pstree and resources

Re-running recap, disabling fdisk, mysql reports/logs, then backing up again:

2018-05-18 11:38:33-05:00 [INFO] --- Starting recap[15964] ---
2018-05-18 11:38:33-05:00 [INFO] -- bash info: 4 4 19 1 release x86_64-unknown-linux-gnu
2018-05-18 11:38:33-05:00 [INFO] recap[15964]: Created lock file: /var/lock/recap.lock
2018-05-18 11:38:33-05:00 [INFO] Starting check for disk space
2018-05-18 11:38:33-05:00 [INFO] Ended check for disk space
2018-05-18 11:38:33-05:00 [INFO] -- Taking backup, storing reports in /var/log/recap/backups
2018-05-18 11:38:33-05:00 [INFO] Starting backup of reports
2018-05-18 11:38:33-05:00 [INFO] Last run was on: 20180518-113812
2018-05-18 11:38:33-05:00 [INFO] Reports found: [ netstat ps pstree resources ]
2018-05-18 11:38:33-05:00 [INFO] Ended backup of reports
2018-05-18 11:38:33-05:00 [INFO] recap[15964]: Caught signal - deleting /var/lock/recap.lock
2018-05-18 11:38:33-05:00 [INFO] Execution time: 0s
2018-05-18 11:38:33-05:00 [INFO] --- Ending recap[15964] ---

Working as intended, i.e. dynamically finding the right reports/logs to act upon.

Testing sending mail functionality

Using the previous config:

...
2018-05-18 11:49:45-05:00 [INFO] -- Report suffix: 20180518-114945
...
2018-05-18 11:49:56-05:00 [INFO] Starting email report
2018-05-18 11:49:56-05:00 [INFO] Finding reports from this run: 20180518-114945
2018-05-18 11:49:56-05:00 [INFO] Reports found: [ netstat ps pstree resources ]
2018-05-18 11:49:56-05:00 [INFO] Ended email report
2018-05-18 11:49:56-05:00 [INFO] Sending email report to: [email protected]
2018-05-18 11:49:56-05:00 [INFO] Sent email report
2018-05-18 11:49:56-05:00 [INFO] recap[17141]: Caught signal - deleting /var/lock/recap.lock
2018-05-18 11:49:56-05:00 [INFO] Execution time: 11s
2018-05-18 11:49:56-05:00 [INFO] --- Ending recap[17141] ---

See it emailed up netsta, ps, pstree, and `resources reports/logs.

Re-running recap, adding back fdisk, mysql reports:

...
2018-05-18 11:50:52-05:00 [INFO] -- Report suffix: 20180518-115052
...
2018-05-18 11:51:03-05:00 [INFO] Starting email report
2018-05-18 11:51:03-05:00 [INFO] Finding reports from this run: 20180518-115052
2018-05-18 11:51:03-05:00 [INFO] Reports found: [ fdisk mysql netstat ps pstree resources ]
2018-05-18 11:51:03-05:00 [INFO] Ended email report
2018-05-18 11:51:03-05:00 [INFO] Sending email report to: [email protected]
2018-05-18 11:51:03-05:00 [INFO] Sent email report
2018-05-18 11:51:03-05:00 [INFO] recap[17342]: Caught signal - deleting /var/lock/recap.lock
2018-05-18 11:51:03-05:00 [INFO] Execution time: 11s
2018-05-18 11:51:03-05:00 [INFO] --- Ending recap[17342] --

Similar as the first test, this is working as intended, dynamically finding the

Testing recaplog

The reports/logs for the day before(YESTERDAYDATE) are mysql, netstat, ps and resources:

2018-05-18 12:27:50-05:00 [INFO] --- Starting recaplog[20400] ---
2018-05-18 12:27:50-05:00 [INFO] recaplog (20400): Created lock file: /var/lock/recaplog.lock
2018-05-18 12:27:50-05:00 [INFO] Compressing old log files
2018-05-18 12:27:50-05:00 [INFO] Finding reports from: 20180517
2018-05-18 12:27:50-05:00 [INFO] Reports found: [ mysql netstat ps resources ]
2018-05-18 12:27:50-05:00 [INFO] Packing mysql...
2018-05-18 12:27:50-05:00 [INFO] Moving 21 logs to: /var/log/recap.test/mysql_daily_20180517
2018-05-18 12:27:50-05:00 [INFO] Compressing 21 logs into: /var/log/recap.test/mysql_daily_20180517.log.tar.gz
2018-05-18 12:27:50-05:00 [INFO] Deleting 21 logs.
2018-05-18 12:27:50-05:00 [INFO] Packing netstat...
2018-05-18 12:27:50-05:00 [INFO] Moving 21 logs to: /var/log/recap.test/netstat_daily_20180517
2018-05-18 12:27:50-05:00 [INFO] Compressing 21 logs into: /var/log/recap.test/netstat_daily_20180517.log.tar.gz
2018-05-18 12:27:50-05:00 [INFO] Deleting 21 logs.
2018-05-18 12:27:50-05:00 [INFO] Packing ps...
2018-05-18 12:27:50-05:00 [INFO] Moving 22 logs to: /var/log/recap.test/ps_daily_20180517
2018-05-18 12:27:50-05:00 [INFO] Compressing 22 logs into: /var/log/recap.test/ps_daily_20180517.log.tar.gz
2018-05-18 12:27:50-05:00 [INFO] Deleting 22 logs.
2018-05-18 12:27:50-05:00 [INFO] Packing resources...
2018-05-18 12:27:50-05:00 [INFO] Moving 22 logs to: /var/log/recap.test/resources_daily_20180517
2018-05-18 12:27:50-05:00 [INFO] Compressing 22 logs into: /var/log/recap.test/resources_daily_20180517.log.tar.gz
2018-05-18 12:27:50-05:00 [INFO] Deleting 22 logs.
2018-05-18 12:27:50-05:00 [INFO] Deleting log files older than 15 days...
2018-05-18 12:27:50-05:00 [INFO] Deleting: 12218 log files.
2018-05-18 12:27:59-05:00 [INFO] Deleting: 0 empty directories.
2018-05-18 12:27:59-05:00 [INFO] recaplog (20400): Caught signal - deleting /var/lock/recaplog.lock
2018-05-18 12:27:59-05:00 [INFO] --- Ending recaplog[20400] ---

@tonyskapunk tonyskapunk mentioned this issue Jun 1, 2018
Merged
@tonyskapunk tonyskapunk modified the milestones: 1.4.0, 2.0.0 Jun 22, 2018
@tonyskapunk
Copy link
Contributor Author

Closing this one since it has been merged into development, yay!

@tonyskapunk tonyskapunk mentioned this issue Jan 9, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
2.0.0
Development

No branches or pull requests
1 participant
@tonyskapunk