ES/tools: add script to create data directory backups on nodes.#306
Open
mgolosova wants to merge 15 commits into
Open
ES/tools: add script to create data directory backups on nodes.#306mgolosova wants to merge 15 commits into
mgolosova wants to merge 15 commits into
Conversation
IATkachenko
suggested changes
Dec 4, 2019
IATkachenko
left a comment
IATkachenko
left a comment
Contributor
There was a problem hiding this comment.
Inline comments in code.
IATkachenko
suggested changes
Dec 6, 2019
Collaborator
Author
|
@IATkachenko, seems like now everything is fixed, please take a look. |
IATkachenko
suggested changes
Dec 12, 2019
Collaborator
Author
|
@IATkachenko, addressed all comments and fixed one more issue popped during tests. |
Collaborator
Author
IATkachenko
previously approved these changes
Dec 13, 2019
We need it at least at `bamboo` node (and thus in this script hardcoded path to the ES data directory is set to the one used on `bamboo`): since it is a development node and we all have full access to all indices, we can erroneously erase something we don't want to be erased and I don't want to dump data from CERN again when it happens. Not sure if it should be applied on the "production" nodes -- at least because we don't have enough disk space there at this moment :)
Better to make sure that all parent directories exist as well.
Logrotate is a proper tool for files rotation, so it's better to use it
instead of inventing another bicycle ;)
With logrotate it is better to use fixed filename (without current date
stamp), so "current" backup file name is changed to `data.tar.gz`.
Due to this change we have to rotate files _before_ creating a new
backup file (else we would never update it -- or rewrite every time and
never rotate).
1-char suffixes indicating the period in backup file name
(`*{w|m|y}.tar.gz`) are replaced with subdirectories for redability
(suffixes would look especially ugly with new `data.tar.gz` base name).
…ctory. If we work with "living" instance of ES, it may do something with data at the moment we are trying to archive them -- and in this case we will have an inconsistent copy of the data directory, which is useless to restore indexed data. Current version is applicable only to the standalone (single-node) ES instance, since in cluster mode every node will create a snapshot in its local directory. To work in this mode, we may need to split the task in two: snapshot update and snapshot backup -- and make sure that the second part won't start before the first one is completed -- or configure 'remote repository' to store whole snapshot on a single node.
This way we have more predictable and expected name for the main (current, latest) backup file (`backup/snapshot.tar.gz`), weekly rotation -- in `weekly/`, monthly -- in `monthly/`, etc.
We don't want to forget to change location/filename somewhere it is used, so it is better to have it defined once and reused everywhere else.
The condition "first week of the month" would be TRUE 7 days in a row, meaning that the first day "monthly/$fname" is found missed, it will be created -- and all the other days logrotate would produce error messages like `ln: creating hard link ...: File exists`. It doesn't look really good and makes me wonder: maybe some time monthly rotation might happen *after* this "first week", and we would end up without monthly copy? So now new copy is created next to the rotation of monthly/yearly copies. And it seems to look better now.
mgolosova
force-pushed
the
es-data-backup
branch
from
December 13, 2019 15:24
93b1ab4 to
d53791e
Compare
Collaborator
Author
|
@IATkachenko, thank you for reviewing the code. I have force-pushed new version, in which "ooops!" commits are squashed with those where "ooops" has happened, and some of commits were reworded: the code has not changed. Please take a (hopefully final) look at the new version. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We need it at least at
bamboonode (and thus in this script hardcodedpath to the ES data directory is set to the one used on
bamboo): sinceit is a development node and we all have full access to all indices, we
can erroneously erase something we don't want to be erased and I don't
want to dump data from CERN again when it happens.
Not sure if it should be applied on the "production" nodes -- at least
because we don't have enough space on them :)