FBUpkeep is a Python library that provides a task executor engine and a command-line tool to execute tasks. Its primary purpose is to run maintenance tasks for Firebird ® servers and databases, but could be easily extended to run other tasks of various type.
Tasks are simple Python classes inherited from Task and registered into fbupkeep_tasks entry point
group (in setup.py).
Tasks are parameterized by the configuration file (using standard configparser Python library module).
Tasks are managed and executed by TaskExecutor which is parametrized using the configuration file and
options that are typically provided as argparse.Namespace.
Module fbupkeep.runner provides fbupkeep command-line utility to work with task executor.
Tasks are executed in named jobs (batch) that run one or more tasks in specified order on single taget (Firebird server or database). Each job could be configured to work with different target.
The fbupkeep script is automatically created in the python scripts directory (it should be in your PATH by default) when you install the library, so you can use it directly.
usage:
fbupkeep [-h] [--version] [--create-config FILENAME]
[--host HOST] [-u USER] [-p PASSWORD]
[-c CONFIG] [-o DIR] [--dry-run]
[-v] [-q] [-l {critical,fatal,error,warn,warning,info,debug,notset}]
[--log-only]
job_name
| -h, --help | show this help message and exit |
| --version | show program’s version number and exit |
| --create-config FILENAME | |
| Create configuration file for job and exit (default: None) | |
| job_name: | Job name |
|---|
| --host HOST | Server host (default: localhost) |
| -u USER, --user USER | |
| User name (default: sysdba) | |
| -p PASSWORD, --password PASSWORD | |
| User password (default: masterkey) | |
| -c CONFIG, --config CONFIG | |
| Configuration file (default: fbupkeep.cfg) | |
| -o DIR, --output-dir DIR | |
| Force directory for log files and other output (default: ${here}/${job_name}) | |
| --dry-run | Prepare execution but do not run tasks (default: False) |
| -v, --verbose | Verbose output (default: False) |
| -q, --quiet | No screen output (default: False) |
| --log-only | Suppress all screen output including error messages (default: False) |
The fbupkeep script can’t work without configuration file. By default, fbupkeep uses the
fbupkeep.cfg configuration file from current working directory, but you can specify
other file by -c or –config command-line option.
Tip
fbupkeep provides –create-config option to create default configuration file for a job.
for example:
fbupkeep --create-config employee.cfg employee
will create file employee.cfg for job employee (i.e. contains section [upkeep_employee])
A configuration file consists of sections, each led by a [section] header, followed by key/value entries separated by = or : string. Section names are case sensitive but keys are not. Leading and trailing whitespace is removed from keys and values. Values can be omitted, in which case the key/value delimiter may also be left out. Values can also span multiple lines, as long as they are indented deeper than the first line of the value.
Important
FBUpkeep uses named jobs. Each job must have it’s own [upkeep_<jobname>] section with configuration for task executor and used tasks.
Individual tasks may use additional sections with their own conventions.
Tip
The convenient way is to use the database name as a job name. For example job name employee for employee database.
Configuration files may include comments, prefixed by # and ; characters. Comments may appear on their own on an otherwise empty line, possibly indented.
Important
The backslash is a special escape character in Python, so it must be doubled if you want to use backslash itself (for example in Windows path specifications).
The DEFAULT section is a special section. If configuration options is not found in specific section, its looked up in DEFAULT section, and if foud there, the value defined there is used.
On execution fbupkeep script automatically adds next options to the default section:
job_name = [job_name option value]
here = [current working directory]
host = [--host option value, default: localhost]
user = [--user option value, default: 'ISC_USER' env. variable or 'sysdba']
password = [--password option value, default: 'ISC_PASSWORD' env. variable or 'masterkey']
output_dir = [--output-dir option value, default: ${here}/${job_name}]
filename = ${job_name}-%Y%m%d
You can freely use these options for interpolation in your configuration.
If you define any from these options in default section, its value would be ignored (replaced by command-line option value).
Values can contain ${section:option} format strings which refer to other values. If the section: part is omitted, interpolation defaults to the current section (and possibly the default values from the special DEFAULT section). Interpolation can span multiple levels.
For example:
[My Section]
foodir: ${dir}/whatever
dir=frob
long: this value continues
in the next line
would resolve the ${dir} to the value of dir (frob in this case). All reference expansions are done on demand.
Important
Values for some options (typically filenames) support datetime.datetime.strftime()
format strings that is interpolated using batch (job) or task execution time. See
comments in generated configuration file for details.
The configuration file created using –create-config option contains the [DEFAULT] and [upkeep_<jobname>] sections, and logging configuration. However, it could not be used without adjustments. If you’ll try to use it as is, you’ll get next error message:
Unexpected error: The configuration does not define a value for the required option 'tasks'
That’s because created configuration file is just a stub where all options and values are commented out. You must update the configuration and provide values for all options that are marked as [REQUIRED].
Important
All jobs must define tasks option with list of tasks to be executed. Tasks are executed in specified order.
You must define values for all options for executor and used tasks that are [REQUIRED] and their value is <UNDEFINED>. Do not forget to uncomment the line with option definition.
Tip
The –dry-run option is provided for convenience to check the configuration without actually running defined tasks. If you would also provide –verbose option, the fbupkeep also prints the configuration options and their values that would be used on execution.
Note
Some configuration options are used by more than one task.
Task executor uses next configuration options:
| tasks: | [REQUIRED] Which tasks should be executed for this job. Tasks are executed in specified order. |
|---|---|
| stop_on_error: | Abort execution on any error (yes) or skip the failing task and continue running (no). [default: yes] |
| use_batch_time: | Time used for filename interpolation, batch start time (yes) or task start time (no). [default: yes] |
Next file was generated by:
fbupkeep --create-config example.cfg example
example.cfg:
; ===========================
; FBUPKEEP configuration file
; ===========================
;
; A configuration file consists of sections, each led by a [section] header, followed by
; key/value entries separated by = or : string. Section names are case sensitive but keys
; are not. Leading and trailing whitespace is removed from keys and values. Values can be
; omitted, in which case the key/value delimiter may also be left out. Values can also
; span multiple lines, as long as they are indented deeper than the first line of the value.
;
; Configuration files may include comments, prefixed by # and ; characters. Comments may
; appear on their own on an otherwise empty line, possibly indented.
;
; Values can contain ${section:option} format strings which refer to other values.
; If the section: part is omitted, interpolation defaults to the current section (and possibly
; the default values from the special DEFAULT section). Interpolation can span multiple levels.
;
; FBUPKEEP uses named jobs. Each job must have it's own [upkeep_<jobname>] section with
; configuration for task executor and used tasks.
;
; Individual tasks may use additional sections with their own conventions.
;
; Logging is configured in separate sections at the end of this file.
; For details see https://docs.python.org/3/howto/logging.html#configuring-logging and
; https://docs.python.org/3/library/logging.config.html#logging-config-fileformat
;
[DEFAULT]
; ==========================
; Section for default values
; ==========================
;
; On execution FBUPKEEP automatically adds next options to the default section:
; job_name = [job_name option value]
; here = [current working directory]
; host = [--host option value, default: localhost]
; user = [--user option value, default: 'ISC_USER' or 'sysdba']
; password = [--password option value, default: 'ISC_PASSWORD' or 'masterkey']
; output_dir = [--output-dir option value, default: ${here}/${job_name}]
; filename = ${job_name}-%Y%m%d
;
; You can freely use these options for interpolation in your configuration.
;
; If you define any from these options in default section, its value would be ignored
; (replaced by command-line option value).
[upkeep_example]
; ===============================
; Configuration for 'example' job
; ===============================
; ---------------------------
; Task executor configuration
; ---------------------------
; tasks
; -----
;
; data type: list
;
; [REQUIRED] Which tasks should be executed for this job.
; Tasks are executed in specified order.
;
; Installed tasks:
; gbak = Create logical (gbak) database backup.
; gbak_restore = Restore database from logical (gbak) backup.
; gstat = Save gstat statistics to file.
; idx_rebuild = Rebuild all user indices in database.
; idx_recompute = Recompute statistics of all database indices.
; remove_old = Remove old files.
; sweep = Perform database sweep operation.
;
;tasks = <UNDEFINED>
; stop_on_error
; -------------
;
; data type: bool
;
; [optional] Abort execution on any error (yes) or skip the failing task and continue running (no).
;
;stop_on_error = yes
; use_batch_time
; --------------
;
; data type: bool
;
; [optional] Time used for filename interpolation, batch start time (yes) or task start time (no).
;
;use_batch_time = yes
; ------------------------------
; Options used by multiple tasks
; ------------------------------
; database
; --------
;
; data type: str
;
; [REQUIRED] Database specification (path or alias)
;
; Used by tasks: gbak, gbak_restore, gstat, idx_rebuild, idx_recompute, sweep
;
;database = <UNDEFINED>
; host
; ----
;
; data type: str
;
; [optional] Firebird server host
;
; Used by tasks: gbak, gbak_restore, gstat, idx_rebuild, idx_recompute, sweep
;
;host = <UNDEFINED>
; user
; ----
;
; data type: str
;
; [REQUIRED] User name to access Firebird service manager
;
; Used by tasks: gbak, gbak_restore, gstat, idx_rebuild, idx_recompute, sweep
;
;user = SYSDBA
; password
; --------
;
; data type: str
;
; [REQUIRED] Password to access Firebird service manager
;
; Used by tasks: gbak, gbak_restore, gstat, idx_rebuild, idx_recompute, sweep
;
;password = masterkey
; gbak_dir
; --------
;
; data type: str
;
; [REQUIRED] Directory for backup files
;
; Used by tasks: gbak, gbak_restore
;
;gbak_dir = <UNDEFINED>, proposed value: ${output_dir}/backup
; gbak_filename
; -------------
;
; data type: str
;
; [REQUIRED] Backup filename template (supports strftime directives)
;
; Used by tasks: gbak, gbak_restore
;
;gbak_filename = <UNDEFINED>, proposed value: ${filename}.fbk
; -----------
; Task 'gbak'
; -----------
; gbak_gc
; -------
;
; data type: bool
;
; [optional] Whether backup should perform garbage collection
;
;gbak_gc = yes
; gbak_logname
; ------------
;
; data type: str
;
; [optional] Backup log filename template (supports strftime directives)
; If not specified, backup log is not created.
;
;gbak_logname = <UNDEFINED>
; -------------------
; Task 'gbak_restore'
; -------------------
; gbak_restore_dir
; ----------------
;
; data type: str
;
; [REQUIRED] Directory for restored databases
;
;gbak_restore_dir = <UNDEFINED>, proposed value: ${output_dir}/restore
; gbak_restore_filename
; ---------------------
;
; data type: str
;
; [REQUIRED] Restored database filename template (supports strftime directives)
;
;gbak_restore_filename = <UNDEFINED>, proposed value: ${filename}.fdb
; gbak_restore_logname
; --------------------
;
; data type: str
;
; [optional] Restore log filename template (supports strftime directives)
; If not specified, restore log is not created.
;
;gbak_restore_logname = <UNDEFINED>
; ------------
; Task 'gstat'
; ------------
; gstat_dir
; ---------
;
; data type: str
;
; [REQUIRED] Directory for output files
;
;gstat_dir = <UNDEFINED>, proposed value: ${output_dir}/gstat
; gstat_filename
; --------------
;
; data type: str
;
; [REQUIRED] Base filename template (supports strftime directives)
;
;gstat_filename = <UNDEFINED>, proposed value: ${filename}.gstat
; ------------------
; Task 'idx_rebuild'
; ------------------
; This task has no private options (not shared with other tasks).
; --------------------
; Task 'idx_recompute'
; --------------------
; This task has no private options (not shared with other tasks).
; -----------------
; Task 'remove_old'
; -----------------
; remove_old_from
; ---------------
;
; data type: list
;
; [REQUIRED] Comma separated list of configuration
; section names. Files are removed according to configuration sections.
; Each section must have next options:
; path: Directory for processing
; files_expire_after: Number of days to keep files, older files could be removed
;
; Optional options:
; pattern: Filename pattern (supports Unix shell-style wildcards), [default: *]
; recursive: yes/no, [default:no]
;
;remove_old_from = <UNDEFINED>
; ------------
; Task 'sweep'
; ------------
; This task has no private options (not shared with other tasks).
; =====================
; Logging configuration
; =====================
;
; For details see https://docs.python.org/3/howto/logging.html#configuring-logging and
; https://docs.python.org/3/library/logging.config.html#logging-config-fileformat
[loggers]
keys=root
[handlers]
keys=root_handler
[formatters]
keys=root_formatter
[logger_root]
level=NOTSET
handlers=root_handler
[handler_root_handler]
; This sends logging output to file in current working directory
class=FileHandler
args=('fbupkeep.log', 'a')
;
; This sends logging output to STDERR
;class=StreamHandler
;args=(sys.stderr,)
;
formatter=root_formatter
[formatter_root_formatter]
format=%(asctime)s %(levelname)s: %(message)s
The fbupkeep library uses logging Python library module. By default, the logging
level is set to WARNING, messages are stored in fbupkeep.log file in current
working directory, and have <timestamp> <level>: <message> format.
For details how to change logging configuration see https://docs.python.org/3/howto/logging.html#configuring-logging and https://docs.python.org/3/library/logging.config.html#logging-config-fileformat
Task that creates logical backup of the database.
Important
This tasks never overwrites backup and log files. In case that output filename already exists in output directory, it’s renamed to filename.<number> (starting from 1) that does not exists yet.
Configuration options:
| database: | [REQUIRED] Database specification (path or alias) |
|---|---|
| gbak_dir: | [REQUIRED] Directory for backup files |
| gbak_filename: | [REQUIRED] Backup filename template (supports strftime directives) |
| gbak_gc: | Whether backup should perform garbage collection (yes/no) |
| gbak_logname: | Backup log filename template (supports strftime directives). If not specified, the backup log is not created. |
| host: | Firebird server host |
| user: | User name to access Firebird service manager |
| password: | Password to access Firebird service manager |
Example:
[upkeep_mydatabase]
; This configuration stores backup as mydatabase-YYYYMMDD.fbk file in d:\mydatabase\backup
; directory along with backup logs. Because we plan to run sweep before backup, it's not
; necessary to perform garbage collection.
;
output_dir = d:\\mydatabase
database = C:\\Data\\mydatabase.fdb
gbak_dir = ${output_dir}\\backup
gbak_filename = ${filename}.fbk
gbak_logname = ${filename}.log
gbak_gc = no
Task that restores database from logical backup. This task is intended to run after gbak task to verify that database could be successfully restored.
Important
This tasks never overwrites database and log files. In case that output filename already exists in output directory, it’s renamed to filename.<number> (starting from 1) that does not exists yet.
Note
The restored database is not deleted. You can use remove_old task to delete it.
Important
When you use time-interpolated filenames for backups, make sure that gbak and gbak_restore tasks use the same timetamp (see use_batch_time option) or other means to ensure that backup file used by gbak_restore exists.
Configuration options:
| database: | [REQUIRED] Database specification (path or alias) |
|---|---|
| gbak_dir: | [REQUIRED] Directory for backup files |
| gbak_filename: | [REQUIRED] Backup filename template (supports strftime directives) |
| gbak_restore_dir: | |
| [REQUIRED] Directory for restored databases | |
| gbak_restore_filename: | |
| [REQUIRED] Restored database filename template (supports strftime directives) | |
| gbak_restore_logname: | |
| Restore log filename template (supports strftime directives). If not specified, the restore log is not created. | |
| host: | Firebird server host |
| user: | User name to access Firebird service manager |
| password: | Password to access Firebird service manager |
Example:
[upkeep_mydatabase]
; This configuration uses backup in mydatabase-YYYYMMDD.fbk file in d:\mydatabase\backup
; directory, and creates database mydatabase-YYYYMMDD.fdb file in d:\mydatabase\restore
; directory along with restore log.
;
output_dir = d:\\mydatabase
gbak_dir = ${output_dir}\\backup
gbak_filename = ${filename}.fbk
gbak_restore_dir = ${output_dir}\\restore
gbak_restore_filename = ${filename}.fdb
gbak_restore_logname = ${filename}.log
Task that performs database sweep operation.
Configuration options:
| database: | [REQUIRED] Database specification (path or alias) |
|---|---|
| host: | Firebird server host |
| user: | User name to access Firebird service manager |
| password: | Password to access Firebird service manager |
Example:
[upkeep_mydatabase]
;
database = C:\\Data\\mydatabase.fdb
Task that collects gstat database statistics.
Important
This tasks never overwrites output files. In case that output filename already exists in output directory, it’s renamed to filename.<number> (starting from 1) that does not exists yet.
Configuration options:
| database: | [REQUIRED] Database specification (path or alias) |
|---|---|
| gstat_dir: | [REQUIRED] Directory for output files |
| gstat_filename: | [REQUIRED] Base filename template (supports strftime directives) |
| host: | Firebird server host |
| user: | User name to access Firebird service manager |
| password: | Password to access Firebird service manager |
Example:
[upkeep_mydatabase]
; This configuration stores database statistics as mydatabase-YYYYMMDD.gstat file in
; d:\mydatabase\gstat directory.
;
output_dir = d:\\mydatabase
database = C:\\Data\\mydatabase.fdb
gstat_dir = ${output_dir}\\gstat
gstat_filename = ${filename}.gstat
Task that recomputes statistics for all database indices.
Configuration options:
| database: | [REQUIRED] Database specification (path or alias) |
|---|---|
| host: | Firebird server host |
| user: | User name to access Firebird service manager |
| password: | Password to access Firebird service manager |
Note
Verbose mode logs & prints each recomputed index.
Example:
[upkeep_mydatabase]
;
database = C:\\Data\\mydatabase.fdb
Task that rebuilds all user (i.e. not system) indices.
Configuration options:
| database: | [REQUIRED] Database specification (path or alias) |
|---|---|
| host: | Firebird server host |
| user: | User name to access Firebird service manager |
| password: | Password to access Firebird service manager |
Example:
[upkeep_mydatabase]
;
database = C:\\Data\\mydatabase.fdb
Note
Verbose mode logs & prints each rebuilt index.
Task that removes old files.
Configuration options:
| remove_old_from: | |
|---|---|
| List of comma-separated section names. | |
Each listed section must/can define next options:
| path: | [REQUIRED] Directory for processing |
|---|---|
| files_expire_after: | |
| [REQUIRED] Number of days to keep files, older files could be removed | |
| pattern: | Filename pattern (supports Unix shell-style wildcards) |
| recursive: | Process path recursively (yes/no) |
Note
Logs & prints each file removed.
Example:
[upkeep_mydatabase]
; Next configuration removes old backups, and database file restored in this job run.
;
remove_old_from = mydatabase_old_backup, mydatabase_old_restore
; Next options are used by gbak and gbak_restore
gbak_dir = ${output_dir}\\backup
gbak_restore_dir = ${output_dir}\\restore
gbak_restore_filename = ${filename}.fdb
[mydatabase_old_backup]
; Remove backups (but keep other files in backup directory) older than 14 days
path = ${upkeep_mydatabase:gbak_dir}
files_expire_after = 14
pattern = *.fbk
recursive = no
[mydatabase_old_restore]
; Remove database previously restored by gbak_restore (if any)
path = ${upkeep_mydatabase:gbak_restore_dir}
; Make sure that we will pass the expiration test
files_expire_after = -2
; We want to remove exactly this database file restored previously by gbak_restore
pattern = ${upkeep_mydatabase:gbak_restore_filename}
recursive = no