When a scrub completes or is cancelled, statistics are updated for
reporting in a later btrfs scrub status command and for resuming the
scrub. Most statistics (such as bytes scrubbed) are additive so scrub
adds the statistics from the current run to the saved statistics.
However, the last_physical statistic is not additive. The value from the
current run should replace the saved value. The current code incorrectly
adds the last_physical from the current run to the previous saved value.
This bug causes the resume point to be incorrectly recorded, so large
areas of the disk are skipped when the scrub resumes. As an example,
assume a disk had 1000000 bytes and scrub was cancelled and resumed each
time 10% (100000 bytes) had been scrubbed.
Run | Start byte | bytes scrubbed | kernel last_physical | saved last_physical
1 | 0 | 100000 | 100000 | 100000
2 | 100000 | 100000 | 200000 | 300000
3 | 300000 | 100000 | 400000 | 700000
4 | 700000 | 100000 | 800000 | 1500000
5 | 1500000 | 0 | immediately completes| completed
In this example, only 40% of the disk is actually scrubbed.
This patch changes the saved/displayed last_physical to track the last
reported value from the kernel.
Signed-off-by: Graham R. Cobb <g.btrfs@cobb.uk.net>
Signed-off-by: David Sterba <dsterba@suse.com>
Since the commmit 8dd3e5dc2d
("btrfs-progs: tests: fix misc-tests/029 to run on NFS") added the
compatibility of NFS, it called run_mayfail() in the last of the test.
However, run_mayfail() always return the original code. If the test
case is not running on NFS, the last `run_mayfail rmdir "$SUBVOL_MNT"`
will fail with return value 1 then the test fails:
================================================================
====== RUN MAYFAIL rmdir btrfs-progs/tests/misc-tests/029-send-p-different-mountpoints/subvol_mnt
rmdir: failed to remove 'btrfs-progs/tests/misc-tests/029-send-p-different-mountpoints/subvol_mnt': No such file or director
failed (ignored, ret=1): rmdir btrfs-progs/tests/misc-tests/029-send-p-different-mountpoints/subvol_mnt
test failed for case 029-send-p-different-mountpoints
=================================================================
Every instrument in this script handles its error well, so do exit 0
manually in the last.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=202645
Fixes: 8dd3e5dc2d ("btrfs-progs: tests: fix misc-tests/029 to run on NFS")
Signed-off-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add structures and API for unified output definition and multiple
formatting backends. Currently there's plain text and json.
The format of each row is defined in struct rowspec, selected using a
key and formatted according to the type. There are extended types for
eg. UUID or pretty size, while direct printf format specifiers work too.
Due to different nature of the outputs, the context structure members
are not always used.
* text output mostly uses indentation and formats the name to a given
width
* json output tracks nesting depth and keeps stack of previous groups
(list or array) and how many member have been printed, as the
separators are allowed only between values and must not preced the
group closing bracket
the nesting depth is hardcoded to 16, counting the global group
The API provides functions to print simple values and some helpers to
format more complex structures.
Signed-off-by: David Sterba <dsterba@suse.com>
Global options should be printed right after the command options, but
there could be text following the options. Add a marker that will allow
to order the options before that text.
Signed-off-by: David Sterba <dsterba@suse.com>
This adds a global --format option to request extended output formats
from each command.
We currently only support text mode. Command help reports what
output formats are available for each command. Global help reports
what valid formats are.
If an invalid format is requested, an error is reported and lists the
valid formats.
Each command sets a bitmask that describes which formats it is capable
of outputting. If a globally valid format is requested of a command
that doesn't support it, an error is reported and command usage dumped.
Commands don't need to specify that they support text output. All
commands are required to output text.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
[ use global config instead of passing cmd_context ]
Signed-off-by: David Sterba <dsterba@suse.com>
For options that do not have the long description, the empty string is
required to mark where the options start. Some commands were missing
that.
Signed-off-by: David Sterba <dsterba@suse.com>
Create directory for all sources that can be used by anything that's not
rellated to a relevant kernel part, all common functions, helpers,
utilities that do not fit any other specific category.
The traditional location would be probably lib/ with all things that are
statically linked to the main binaries, but we have libbtrfs and
libbtrfsutil so this would be confusing.
Signed-off-by: David Sterba <dsterba@suse.com>
This is first instance of commands files moving to a separate directory,
that will be cmds/, thus the files can drop the prefix. We can further
split files into specific parts of a given command. The quota file was
selected as the smallest.
Signed-off-by: David Sterba <dsterba@suse.com>
Our build allows easy definition of CFLAGs that apply only to a given
file, like cmds_restore_cflags and cmds-restore.c .
This is done by series of transformations that convert the file name to
a variable name, when that is defined it's used.
To support files in directories outside of the top level we need to
convert the / too. The function 'subst' supports only a single string,
so they have to be nested.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 756105181e ("btrfs-progs: check: supplement extent backref
list with rbtree") changed the backref implementation to use rb tree
and also commented the old implementations. It's been almost 2 years
since that change and it's unlikely the old version will ever be used,
so just remove it.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit ba23855cdc ("btrfs-progs: send: use splice syscall instead of
read/write to transfer buffer") changed the send implementation to use
splice(). The old read/write implementation hasn't be used for at least
3 years, it's time to remove it.
The splice mechanism proved to be reliable and the manual buffer copy
fallback is not needed, besides that splice is probably faster.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This piece of code has been commented since 2009, given the number of
changes that have happened it's unlikely it could be made to work or is
needed at all. Just delete it.
The code was disabled in commit 95d3f20b51 ("Mixed back reference
(FORWARD ROLLING FORMAT CHANGE)") that changed the format significantly
and we don't need the compatibility code anymore.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
'pin' is always true in __free_extent so there is no point in checking
it. Just remove the if and unindent the code.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Prievous patch added throughput and time left calculatios, but with more
information it becomes less clear. Switch to the output format used in
several other commands that prints header, followed by colon, whitespace
and the value. Grouped values are indented by 2 spaces.
This patch uses the space info that is more accurate than the total
size. The used space is what scrub will check, however the multiplicity
is not yet taken into account, so this works only for the 'single'
profile.
Sample output:
UUID: bf8720e0-606b-4065-8320-b48df2e8e669
Scrub started: Fri Jun 14 12:00:00 2019
Status: running
Duration: 0:14:11
Time left: 0:04:04
ETA: Fri Jun 14 12:18:15 2019
Total to scrub: 182.55GiB
Bytes scrubbed: 141.80GiB
Rate: 170.63MiB/s
Error summary: csum=7
Corrected: 0
Uncorrectable: 7
Unverified: 0
For the reference, this is 'fi df':
Data, single: total=261.00GiB, used=179.91GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=5.00GiB, used=2.64GiB
GlobalReserve, single: total=375.23MiB, used=0.00B
Several repeated runs of scrub showed that the time estimate is very
close to the final time (within tens of seconds).
Signed-off-by: David Sterba <dsterba@suse.com>
The table has been updated, copy the changes so that we can utilize it
for cleanups.
Note, ncopies for raid5 and rai6 was wrong and is now correct.
Signed-off-by: David Sterba <dsterba@suse.com>
The estimation is based on the allocated bytes, so it might be
overestimated. Scrub reports the size of all bytes scrubbed, taking
into account the replication, so we're comparing that with total sum
over all devices that we get from DEV_INFO, in the same units.
Example output:
scrub status for xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
scrub started at Fri May 31 15:56:57 2019, running for 0:04:31
total 62.55GiB scrubbed at rate 236.37MiB/s, time left: 0:12:31
no errors found
Pull-request: #177
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
Signed-off-by: David Sterba <dsterba@suse.com>
The command 'subvolume show' would return error code in case quotas are
not enabled or in any other error. In case they're not enabled, it's not
fatal, no-qgroup setups are quite common.
Signed-off-by: David Sterba <dsterba@suse.com>
Print qgroup information one row per value and don't print the
information at all if quotas are not enabled. Sample output:
subv1
Name: subv1
UUID: 58aa0df4-6bde-3e4e-b9f6-631d9c23578f
Parent UUID: -
Received UUID: -
Creation time: 2019-06-19 12:34:56 +0200
Subvolume ID: 258
Generation: 9
Gen at creation: 9
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):
Quota group: 0/258
Limit referenced: -
Limit exclusive: 1.00GiB
Usage referenced: 16.00KiB
Usage exclusive: 16.00KiB
Signed-off-by: David Sterba <dsterba@suse.com>
Most command groups just pass their own command group to
handle_command_group. We can remove the explicit definitions
of command group callbacks by passing the cmd_struct to
handle_command_group and allowing it to resolve the group from it.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Now that every call site has a cmd_struct, we can just pass the cmd_struct
to usage to print the usager information. This allows us to interpret
the format flags we'll add later in this series to inform the user of
which output formats any given command supports.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Now that we have a cmd_struct everywhere, we can pass it to
clean_args_no_options and have it resolve the usage string from
it there. This is necessary for it to pass the cmd_struct to
usage() in the next patch.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Graham R. Cobb