Skip to content

pt-replica-restart

NAME

pt-replica-restart - Watch and restart MySQL replication after errors.

SYNOPSIS

Usage

pt-replica-restart [OPTIONS] [DSN]

pt-replica-restart watches one or more MySQL replication replicas for errors, and tries to restart replication if it stops.

RISKS

Percona Toolkit is mature, proven in the real world, and well tested, but all database tools can pose a risk to the system and the database server. Before using this tool, please:

  • Read the tool’s documentation

  • Review the tool’s known “BUGS”

  • Test the tool on a non-production server

  • Backup your production server and verify the backups

DESCRIPTION

pt-replica-restart watches one or more MySQL replication replicas and tries to skip statements that cause errors. It polls replicas intelligently with an exponentially varying sleep time. You can specify errors to skip and run the replicas until a certain binlog position.

Although this tool can help a replica advance past errors, you should not rely on it to “fix” replication. If replica errors occur frequently or unexpectedly, you should identify and fix the root cause.

OUTPUT

pt-replica-restart prints a line every time it sees the replica has an error. By default this line is: a timestamp, connection information, relay_log_file, relay_log_pos, and last_errno. You can add more information using the --verbose option. You can suppress all output using the --quiet option.

SLEEP

pt-replica-restart sleeps intelligently between polling the replica. The current sleep time varies.

  • The initial sleep time is given by --sleep.

  • If it checks and finds an error, it halves the previous sleep time.

  • If it finds no error, it doubles the previous sleep time.

  • The sleep time is bounded below by --min-sleep and above by --max-sleep.

  • Immediately after finding an error, pt-replica-restart assumes another error is very likely to happen next, so it sleeps the current sleep time or the initial sleep time, whichever is less.

GLOBAL TRANSACTION IDS

As of Percona Toolkit 2.2.8, pt-replica-restart supports Global Transaction IDs introduced in MySQL 5.6.5. It’s important to keep in mind that:

  • pt-replica-restart will not skip transactions when multiple replication threads are being used (replica_parallel_workers > 0). pt-replica-restart does not know what the GTID event is of the failed transaction of a specific replica thread.

  • The default behavior is to skip the next transaction from the replica’s source. Writes can originate on different servers, each with their own UUID.

    See --source-uuid.

EXIT STATUS

An exit status of 0 (sometimes also called a return value or return code) indicates success. Any other value represents the exit status of the Perl process itself, or of the last forked process that exited if there were multiple servers to monitor.

COMPATIBILITY

pt-replica-restart should work on many versions of MySQL. Lettercase of many output columns from SHOW REPLICA STATUS has changed over time, so it treats them all as lowercase.

OPTIONS

This tool accepts additional command-line arguments. Refer to the “SYNOPSIS” and usage information for details.

--always

Start replicas even when there is no error. With this option enabled, pt-replica-restart will not let you stop the replica manually if you want to!

--ask-pass

Prompt for a password when connecting to MySQL.

--charset

short form: -A; type: string

Default character set. If the value is utf8, sets Perl’s binmode on STDOUT to utf8, passes the mysql_enable_utf8 option to DBD::mysql, and runs SET NAMES UTF8 after connecting to MySQL. Any other value sets binmode on STDOUT without the utf8 layer, and runs SET NAMES after connecting to MySQL.

--[no]check-relay-log

default: yes

Check the last relay log file and position before checking for replica errors.

By default pt-replica-restart will not doing anything (it will just sleep) if neither the relay log file nor the relay log position have changed since the last check. This prevents infinite loops (i.e. restarting the same error in the same relay log file at the same relay log position).

For certain replica errors, however, this check needs to be disabled by specifying --no-check-relay-log. Do not do this unless you know what you are doing!

--config

type: Array

Read this comma-separated list of config files; if specified, this must be the first option on the command line.

--daemonize

Fork to the background and detach from the shell. POSIX operating systems only.

--database

short form: -D; type: string

Database to use.

--defaults-file

short form: -F; type: string

Only read mysql options from the given file. You must give an absolute pathname.

--error-length

type: int

Max length of error message to print. When --verbose is set high enough to print the error, this option will truncate the error text to the specified length. This can be useful to prevent wrapping on the terminal.

--error-numbers

type: hash

Only restart this comma-separated list of errors. Makes pt-replica-restart only try to restart if the error number is in this comma-separated list of errors. If it sees an error not in the list, it will exit.

The error number is in the last_errno column of SHOW REPLICA STATUS.

--error-text

type: string

Only restart errors that match this pattern. A Perl regular expression against which the error text, if any, is matched. If the error text exists and matches, pt-replica-restart will try to restart the replica. If it exists but doesn’t match, pt-replica-restart will exit.

The error text is in the last_error column of SHOW REPLICA STATUS.

--help

Show help and exit.

--host

short form: -h; type: string

Connect to host.

--log

type: string

Print all output to this file when daemonized.

--master-uuid

type: string

This option is deprecated and will be removed in future releases. Use --source-user instead.

--max-sleep

type: float; default: 64

Maximum sleep seconds.

The maximum time pt-replica-restart will sleep before polling the replica again. This is also the time that pt-replica-restart will wait for all other running instances to quit if both --stop and --monitor are specified.

See “SLEEP”.

--min-sleep

type: float; default: 0.015625

The minimum time pt-replica-restart will sleep before polling the replica again. See “SLEEP”.

--monitor

Whether to monitor the replica (default). Unless you specify –monitor explicitly, --stop will disable it.

--password

short form: -p; type: string

Password to use when connecting. If password contains commas they must be escaped with a backslash: “exam,ple”

--pid

type: string

Create the given PID file. The tool won’t start if the PID file already exists and the PID it contains is different than the current PID. However, if the PID file exists and the PID it contains is no longer running, the tool will overwrite the PID file with the current PID. The PID file is removed automatically when the tool exits.

--port

short form: -P; type: int

Port number to use for connection.

--quiet

short form: -q

Suppresses normal output (disables --verbose).

--recurse

type: int; default: 0

Watch replicas of the specified server, up to the specified number of servers deep in the hierarchy. The default depth of 0 means “just watch the replica specified.”

pt-replica-restart examines SHOW PROCESSLIST and tries to determine which connections are from replicas, then connect to them. See --recursion-method.

Recursion works by finding all replicas when the program starts, then watching them. If there is more than one replica, pt-replica-restart uses fork() to monitor them.

This also works if you have configured your replicas to show up in SHOW REPLICAS. The minimal configuration for this is the report_host parameter, but there are other “report” parameters as well for the port, username, and password.

--recursion-method

type: array; default: processlist,hosts

Preferred recursion method used to find replicas.

Possible methods are:

METHOD       USES
===========  ==================
processlist  SHOW PROCESSLIST
hosts        SHOW REPLICAS (SHOW SLAVE HOSTS before MySQL 8.1)
none         Do not find replicas

The processlist method is preferred because SHOW REPLICAS is not reliable. However, the hosts method is required if the server uses a non-standard port (not 3306). Usually pt-replica-restart does the right thing and finds the replicas, but you may give a preferred method and it will be used first. If it doesn’t find any replicas, the other methods will be tried.

--run-time

type: time

Time to run before exiting. Causes pt-replica-restart to stop after the specified time has elapsed. Optional suffix: s=seconds, m=minutes, h=hours, d=days; if no suffix, s is used.

--sentinel

type: string; default: /tmp/pt-replica-restart-sentinel

Exit if this file exists.

--slave-user

type: string

This option is deprecated and will be removed in future releases. Use --replica-user instead.

--slave-password

type: string

This option is deprecated and will be removed in future releases. Use --replica-password instead.

--replica-user

type: string

Sets the user to be used to connect to the replicas. This parameter allows you to have a different user with less privileges on the replicas but that user must exist on all replicas.

--replica-password

type: string

Sets the password to be used to connect to the replicas. It can be used with –replica-user and the password for the user must be the same on all replicas.

--set-vars

type: Array

Set the MySQL variables in this comma-separated list of variable=value pairs.

By default, the tool sets:

wait_timeout=10000

Variables specified on the command line override these defaults. For example, specifying --set-vars wait_timeout=500 overrides the defaultvalue of 10000.

The tool prints a warning and continues if a variable cannot be set.

--skip-count

type: int; default: 1

Number of statements to skip when restarting the replica.

--source-uuid

type: string

When using GTID, an empty transaction should be created in order to skip it. If writes are coming from different nodes in the replication tree above, it is not possible to know which event from which UUID to skip.

By default, transactions from the replica’s source ('Source_UUID' from SHOW REPLICA STATUS) are skipped.

For example, with

source1 -> replica1 -> replica2

When skipping events on replica2 that were written to source1, you must specify the UUID of source1, else the tool will use the UUID of replica1 by default.

See “GLOBAL TRANSACTION IDS”.

--sleep

type: int; default: 1

Initial sleep seconds between checking the replica.

See “SLEEP”.

--socket

short form: -S; type: string

Socket file to use for connection.

--stop

Stop running instances by creating the sentinel file.

Causes pt-replica-restart to create the sentinel file specified by --sentinel. This should have the effect of stopping all running instances which are watching the same sentinel file. If --monitor isn’t specified, pt-replica-restart will exit after creating the file. If it is specified, pt-replica-restart will wait the interval given by --max-sleep, then remove the file and continue working.

You might find this handy to stop cron jobs gracefully if necessary, or to replace one running instance with another. For example, if you want to stop and restart pt-replica-restart every hour (just to make sure that it is restarted every hour, in case of a server crash or some other problem), you could use a crontab line like this:

0 * * * * :program:`pt-replica-restart` --monitor --stop --sentinel /tmp/pt-replica-restartup

The non-default --sentinel will make sure the hourly cron job stops only instances previously started with the same options (that is, from the same cron job).

See also --sentinel.

--until-master

type: string

This option is deprecated and will be removed in future releases. Use --until-source instead.

--until-source

type: string

Run until this source binary log file and position. Start the replica, and retry if it fails, until it reaches the given replication coordinates. The coordinates are the logfile and position on the source, given by relay_source_log_file, exec_source_log_pos. The argument must be in the format “file,pos”. Separate the filename and position with a single comma and no space.

This will also cause an UNTIL clause to be given to START REPLICA.

After reaching this point, the replica should be stopped and pt-replica-restart will exit.

--until-relay

type: string

Run until this relay log file and position. Like --until-source, but in the replica’s relay logs instead. The coordinates are given by relay_log_file, relay_log_pos.

--user

short form: -u; type: string

User for login if not current user.

--verbose

short form: -v; cumulative: yes; default: 1

Adds more information to the output. This flag can be specified multiple times. e.g. -v -v OR -vv. By default (no verbose flag) the tool outputs connection information, a timestamp, relay_log_file, relay_log_pos, and last_errno. One flag (-v) adds last_error. See also --error-length. Two flags (-vv) prints the current sleep time each time pt-replica-restart sleeps. To suppress all output use the --quiet option.

--version

Show version and exit.

--[no]version-check

default: yes

Check for the latest version of Percona Toolkit, MySQL, and other programs.

This is a standard “check for updates automatically” feature, with two additional features. First, the tool checks its own version and also the versions of the following software: operating system, Percona Monitoring and Management (PMM), MySQL, Perl, MySQL driver for Perl (DBD::mysql), and Percona Toolkit. Second, it checks for and warns about versions with known problems. For example, MySQL 5.5.25 had a critical bug and was re-released as 5.5.25a.

A secure connection to Percona’s Version Check database server is done to perform these checks. Each request is logged by the server, including software version numbers and unique ID of the checked system. The ID is generated by the Percona Toolkit installation script or when the Version Check database call is done for the first time.

Any updates or known problems are printed to STDOUT before the tool’s normal output. This feature should never interfere with the normal operation of the tool.

For more information, visit https://www.percona.com/doc/percona-toolkit/LATEST/version-check.html.

Show version and exit.

DSN OPTIONS

These DSN options are used to create a DSN. Each option is given like option=value. The options are case-sensitive, so P and p are not the same option. There cannot be whitespace before or after the = and if the value contains whitespace it must be quoted. DSN options are comma-separated. See the percona-toolkit manpage for full details.

  • A

dsn: charset; copy: yes

Default character set.

  • D

dsn: database; copy: yes

Default database.

  • F

dsn: mysql_read_default_file; copy: yes

Only read default options from the given file

  • h

dsn: host; copy: yes

Connect to host.

  • p

dsn: password; copy: yes

Password to use when connecting. If password contains commas they must be escaped with a backslash: “exam,ple”

  • P

dsn: port; copy: yes

Port number to use for connection.

  • S

dsn: mysql_socket; copy: yes

Socket file to use for connection.

  • u

dsn: user; copy: yes

User for login if not current user.

  • s

dsn: mysql_ssl; copy: yes

Create SSL connection

ENVIRONMENT

The environment variable PTDEBUG enables verbose debugging output to STDERR. To enable debugging and capture all output to a file, run the tool like:

PTDEBUG=1 pt-replica-restart ... > FILE 2>&1

Be careful: debugging output is voluminous and can generate several megabytes of output.

ATTENTION

Using <PTDEBUG> might expose passwords. When debug is enabled, all command line parameters are shown in the output.

SYSTEM REQUIREMENTS

You need Perl, DBI, DBD::mysql, and some core packages that ought to be installed in any reasonably new version of Perl.

BUGS

For a list of known bugs, see https://jira.percona.com/projects/PT/issues.

Please report bugs at https://jira.percona.com/projects/PT. Include the following information in your bug report:

  • Complete command-line used to run the tool

  • Tool --version

  • MySQL version of all servers involved

  • Output from the tool including STDERR

  • Input files (log/dump/config files, etc.)

If possible, include debugging output by running the tool with PTDEBUG; see “ENVIRONMENT”.

DOWNLOADING

Visit http://www.percona.com/software/percona-toolkit/ to download the latest release of Percona Toolkit. Or, get the latest release from the command line:

wget percona.com/get/percona-toolkit.tar.gz

wget percona.com/get/percona-toolkit.rpm

wget percona.com/get/percona-toolkit.deb

You can also get individual tools from the latest release:

wget percona.com/get/TOOL

Replace TOOL with the name of any tool.

AUTHORS

Baron Schwartz

ABOUT PERCONA TOOLKIT

This tool is part of Percona Toolkit, a collection of advanced command-line tools for MySQL developed by Percona. Percona Toolkit was forked from two projects in June, 2011: Maatkit and Aspersa. Those projects were created by Baron Schwartz and primarily developed by him and Daniel Nichter. Visit http://www.percona.com/software/ to learn about other free, open-source software from Percona.

VERSION

pt-replica-restart 3.7.0