Create directory for all sources that can be used by anything that's not
rellated to a relevant kernel part, all common functions, helpers,
utilities that do not fit any other specific category.
The traditional location would be probably lib/ with all things that are
statically linked to the main binaries, but we have libbtrfs and
libbtrfsutil so this would be confusing.
Signed-off-by: David Sterba <dsterba@suse.com>
This is first instance of commands files moving to a separate directory,
that will be cmds/, thus the files can drop the prefix. We can further
split files into specific parts of a given command. The quota file was
selected as the smallest.
Signed-off-by: David Sterba <dsterba@suse.com>
Our build allows easy definition of CFLAGs that apply only to a given
file, like cmds_restore_cflags and cmds-restore.c .
This is done by series of transformations that convert the file name to
a variable name, when that is defined it's used.
To support files in directories outside of the top level we need to
convert the / too. The function 'subst' supports only a single string,
so they have to be nested.
Signed-off-by: David Sterba <dsterba@suse.com>
Commit 756105181e ("btrfs-progs: check: supplement extent backref
list with rbtree") changed the backref implementation to use rb tree
and also commented the old implementations. It's been almost 2 years
since that change and it's unlikely the old version will ever be used,
so just remove it.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Commit ba23855cdc ("btrfs-progs: send: use splice syscall instead of
read/write to transfer buffer") changed the send implementation to use
splice(). The old read/write implementation hasn't be used for at least
3 years, it's time to remove it.
The splice mechanism proved to be reliable and the manual buffer copy
fallback is not needed, besides that splice is probably faster.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This piece of code has been commented since 2009, given the number of
changes that have happened it's unlikely it could be made to work or is
needed at all. Just delete it.
The code was disabled in commit 95d3f20b51 ("Mixed back reference
(FORWARD ROLLING FORMAT CHANGE)") that changed the format significantly
and we don't need the compatibility code anymore.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
'pin' is always true in __free_extent so there is no point in checking
it. Just remove the if and unindent the code.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Prievous patch added throughput and time left calculatios, but with more
information it becomes less clear. Switch to the output format used in
several other commands that prints header, followed by colon, whitespace
and the value. Grouped values are indented by 2 spaces.
This patch uses the space info that is more accurate than the total
size. The used space is what scrub will check, however the multiplicity
is not yet taken into account, so this works only for the 'single'
profile.
Sample output:
UUID: bf8720e0-606b-4065-8320-b48df2e8e669
Scrub started: Fri Jun 14 12:00:00 2019
Status: running
Duration: 0:14:11
Time left: 0:04:04
ETA: Fri Jun 14 12:18:15 2019
Total to scrub: 182.55GiB
Bytes scrubbed: 141.80GiB
Rate: 170.63MiB/s
Error summary: csum=7
Corrected: 0
Uncorrectable: 7
Unverified: 0
For the reference, this is 'fi df':
Data, single: total=261.00GiB, used=179.91GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=5.00GiB, used=2.64GiB
GlobalReserve, single: total=375.23MiB, used=0.00B
Several repeated runs of scrub showed that the time estimate is very
close to the final time (within tens of seconds).
Signed-off-by: David Sterba <dsterba@suse.com>
The table has been updated, copy the changes so that we can utilize it
for cleanups.
Note, ncopies for raid5 and rai6 was wrong and is now correct.
Signed-off-by: David Sterba <dsterba@suse.com>
The estimation is based on the allocated bytes, so it might be
overestimated. Scrub reports the size of all bytes scrubbed, taking
into account the replication, so we're comparing that with total sum
over all devices that we get from DEV_INFO, in the same units.
Example output:
scrub status for xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
scrub started at Fri May 31 15:56:57 2019, running for 0:04:31
total 62.55GiB scrubbed at rate 236.37MiB/s, time left: 0:12:31
no errors found
Pull-request: #177
Signed-off-by: Grzegorz Kowal <grzegorz@amuncode.org>
Signed-off-by: David Sterba <dsterba@suse.com>
The command 'subvolume show' would return error code in case quotas are
not enabled or in any other error. In case they're not enabled, it's not
fatal, no-qgroup setups are quite common.
Signed-off-by: David Sterba <dsterba@suse.com>
Print qgroup information one row per value and don't print the
information at all if quotas are not enabled. Sample output:
subv1
Name: subv1
UUID: 58aa0df4-6bde-3e4e-b9f6-631d9c23578f
Parent UUID: -
Received UUID: -
Creation time: 2019-06-19 12:34:56 +0200
Subvolume ID: 258
Generation: 9
Gen at creation: 9
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):
Quota group: 0/258
Limit referenced: -
Limit exclusive: 1.00GiB
Usage referenced: 16.00KiB
Usage exclusive: 16.00KiB
Signed-off-by: David Sterba <dsterba@suse.com>
Most command groups just pass their own command group to
handle_command_group. We can remove the explicit definitions
of command group callbacks by passing the cmd_struct to
handle_command_group and allowing it to resolve the group from it.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Now that every call site has a cmd_struct, we can just pass the cmd_struct
to usage to print the usager information. This allows us to interpret
the format flags we'll add later in this series to inform the user of
which output formats any given command supports.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Now that we have a cmd_struct everywhere, we can pass it to
clean_args_no_options and have it resolve the usage string from
it there. This is necessary for it to pass the cmd_struct to
usage() in the next patch.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch passes the cmd_struct to the command callback function. This
has several purposes: It allows the command callback to identify which
command was used to call it. It also gives us direct access to the
usage associated with that command.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Rather than having global command usage and callbacks used to create
cmd_structs in the command array, establish the cmd_struct structures
separately and use those. The next commit in the series passes the
cmd_struct to the command callbacks such that we can access flags
and determine which of several potential command we were called as.
This establishes several macros to more easily define the commands
within each command's source.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
In preparation to use cmd_struct as the command entry point, we need
to split out the 'filesystem balance' handling to not call cmd_balance
directly. The reason is that the flags that indicate a command is
hidden are a part of cmd_struct and so we can use a cmd_struct as a
direct alias in another command group and ALSO have it be hidden
without declaring another cmd_struct.
This change has no immediate impact since cmd_balance will still
use its usage information directly from cmds-balance.c. It will
take effect once we start passing cmd_structs around for usage
information.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The usage definitions for send and receive follow the command
definitions, which use them. This works because we declare them
in commands.h. When we move to using cmd_struct as the entry point,
these declarations will be removed, breaking the commands. Since
that would be an otherwise unrelated change, this patch reorders
them separately.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We use an int for 'full', 'all', and 'err' when we really mean a boolean.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch reports on the first-level qgroup, if any, associated with
a particular subvolume. It displays the usage and limit, subject
to the usual unit parameters.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The only mechanism we have in the progs for searching qgroups is to load
all of them and filter the results. This works for qgroup show but
to add quota information to 'btrfs subvoluem show' it's pretty wasteful.
This patch splits out setting up the search and performing the search so
we can search for a single qgroupid more easily. Since TREE_SEARCH
will give results that don't strictly match the search terms, we add
a filter to match only the results we care about.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
We use structures to pass the info and limit from the kernel as items
but store the individual values separately in btrfs_qgroup. We already
have a btrfs_qgroup_limit structure that's used for setting the limit.
This patch introduces a btrfs_qgroup_info structure and uses that and
btrfs_qgroup_limit in btrfs_qgroup.
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When compiling btrfs-progs with GCC 9 (9.1.0), we got the following
warnings:
In file included from utils.h:30,
from cmds-receive.c:45:
cmds-receive.c: In function 'process_subvol':
messages.h:42:3: warning: '%s' directive argument is null [-Wformat-overflow=]
42 | __btrfs_error((fmt), ##__VA_ARGS__); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cmds-receive.c:178:3: note: in expansion of macro 'error'
178 | error("subvol: another one already started, path buf: %s",
| ^~~~~
[CC] cmds-inspect-tree-stats.o
cmds-receive.c: In function 'process_snapshot':
messages.h:42:3: warning: '%s' directive argument is null [-Wformat-overflow=]
42 | __btrfs_error((fmt), ##__VA_ARGS__); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cmds-receive.c:248:3: note: in expansion of macro 'error'
248 | error("snapshot: another one already started, path buf: %s",
| ^~~~~
[FIX]
We're using wrong member for the error output.
Fix the member to output.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
gcc 9.1 reports:
utils.c: In function ‘test_num_disk_vs_raid’:
utils.c:1931:3: warning: attribute ‘fallthrough’ not preceding a case label or default label
1931 | __attribute__ ((fallthrough));
| ^~~~~~~~~~~~~
As there's no other label to fall to (default: is the first one), remove the
annotation.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
GCC 9.1.0 will report the following error when compiling btrfs-progs:
In file included from print-tree.c:24:
ctree.h: In function 'btrfs_dev_stats_values':
ctree.h:2408:9: warning: taking address of packed member of 'struct btrfs_dev_stats_item' may result in an unaligned pointer value [-Waddress-of-packed-member]
2408 | return p->values;
| ^
[FIX]
Follow the kernel way of accessing dev stats by using
btrfs_dev_stats_value(eb, ptr, index).
So that we don't need to bother accessing the packed member.
This also unifies the helper function in kernel and btrfs-progs.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add const prefix for the following parameters:
- @eb of memcmp_extent_buffer()
- @eb of read_extent_buffer()
This backports kernel commit 1cbb1f454e53 ("btrfs: struct-funcs,
constify readers") to btrfs-progs.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When btrfs-progs is compiled with D=asan, it can't pass even the very
basic fsck tests due to btrfs-image has memory leak:
=== START TEST /home/adam/btrfs/btrfs-progs/tests//fsck-tests/001-bad-file-extent-bytenr
restoring image default_case.img
=================================================================
==7790==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 104 byte(s) in 1 object(s) allocated from:
#0 0x7f1d3b738389 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cc:86
#1 0x560ca6b7f4ff in btrfs_add_delayed_tree_ref /home/adam/btrfs/btrfs-progs/delayed-ref.c:569
#2 0x560ca6af2d0b in btrfs_free_extent /home/adam/btrfs/btrfs-progs/extent-tree.c:2155
#3 0x560ca6ac16ca in __btrfs_cow_block /home/adam/btrfs/btrfs-progs/ctree.c:319
#4 0x560ca6ac1d8c in btrfs_cow_block /home/adam/btrfs/btrfs-progs/ctree.c:383
#5 0x560ca6ac6c8e in btrfs_search_slot /home/adam/btrfs/btrfs-progs/ctree.c:1153
#6 0x560ca6ab7e83 in fixup_device_size image/main.c:2113
#7 0x560ca6ab9279 in fixup_chunks_and_devices image/main.c:2333
#8 0x560ca6ab9ada in restore_metadump image/main.c:2455
#9 0x560ca6abaeba in main image/main.c:2723
#10 0x7f1d3b148ce2 in __libc_start_main (/usr/lib/libc.so.6+0x23ce2)
... tons of similar leakage for delayed_tree_ref ...
Direct leak of 96 byte(s) in 1 object(s) allocated from:
#0 0x7f1d3b738389 in __interceptor_malloc /build/gcc/src/gcc/libsanitizer/asan/asan_malloc_linux.cc:86
#1 0x560ca6b7f5fb in btrfs_add_delayed_tree_ref /home/adam/btrfs/btrfs-progs/delayed-ref.c:583
#2 0x560ca6af5679 in alloc_tree_block /home/adam/btrfs/btrfs-progs/extent-tree.c:2503
#3 0x560ca6af57ac in btrfs_alloc_free_block /home/adam/btrfs/btrfs-progs/extent-tree.c:2524
#4 0x560ca6ac115b in __btrfs_cow_block /home/adam/btrfs/btrfs-progs/ctree.c:290
#5 0x560ca6ac1d8c in btrfs_cow_block /home/adam/btrfs/btrfs-progs/ctree.c:383
#6 0x560ca6b7bb15 in commit_tree_roots /home/adam/btrfs/btrfs-progs/transaction.c:98
#7 0x560ca6b7c525 in btrfs_commit_transaction /home/adam/btrfs/btrfs-progs/transaction.c:192
#8 0x560ca6ab92be in fixup_chunks_and_devices image/main.c:2337
#9 0x560ca6ab9ada in restore_metadump image/main.c:2455
#10 0x560ca6abaeba in main image/main.c:2723
#11 0x7f1d3b148ce2 in __libc_start_main (/usr/lib/libc.so.6+0x23ce2)
... tons of similar leakage for delayed_ref_head ...
SUMMARY: AddressSanitizer: 1600 byte(s) leaked in 16 allocation(s).
failed to restore image ./default_case.img
[CAUSE]
Commit c6039704c5 ("btrfs-progs: Add delayed refs infrastructure")
introduces delayed ref infrastructure for free space tree, however the
refcount_dec_and_test() from kernel code is wrongly backported.
refcount_dec_and_test() will return true if the refcount reaches 0.
So kernel code will free the allocated space as expected:
if (refcount_dec_and_test(&ref->refs)) {
kmem_cache_free();
}
However btrfs-progs backport is using the opposite condition:
if (--ref->refs) {
kfree();
}
This will not free the memory for the last user, but for refs >= 2.
Causing both use-after-free and memory leak for any offline write
operation.
[FIX]
Fix the (--ref->refs) condition to (--ref->refs == 0) to fix the
backport error.
Fixes: c6039704c5 ("btrfs-progs: Add delayed refs infrastructure")
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
There is a bug report of unexpected ENOSPC from btrfs-convert, issue #123.
After some debugging, even when we have enough unallocated space, we
still hit ENOSPC at btrfs_reserve_extent().
[CAUSE]
Btrfs-progs relies on chunk preallocator to make enough space for
data/metadata.
However after the introduction of delayed-ref, it's no longer reliable
to rely on btrfs_space_info::bytes_used and
btrfs_space_info::bytes_pinned to calculate used metadata space.
For a running transaction with a lot of allocated tree blocks,
btrfs_space_info::bytes_used stays its original value, and will only be
updated when running delayed ref.
This makes btrfs-progs chunk preallocator completely useless. And for
btrfs-convert/mkfs.btrfs --rootdir, if we're going to have enough
metadata to fill a metadata block group in one transaction, we will hit
ENOSPC no matter whether we have enough unallocated space.
[FIX]
This patch will introduce btrfs_space_info::bytes_reserved to track how
many space we have reserved but not yet committed to extent tree.
To support this change, this commit also introduces the following
modification:
- More comment on btrfs_space_info::bytes_*
To make code a little easier to read
- Export update_space_info() to preallocate empty data/metadata space
info for mkfs.
For mkfs, we only have a temporary fs image with SYSTEM chunk only.
Export update_space_info() so that we can preallocate empty
data/metadata space info before we start a transaction.
- Proper btrfs_space_info::bytes_reserved update
The timing is the as kernel (except we don't need to update
bytes_reserved for data extents)
* Increase bytes_reserved when call alloc_reserved_tree_block()
* Decrease bytes_reserved when running delayed refs
With the help of head->must_insert_reserved to determine whether we
need to decrease.
Issue: #123
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This patch will export disk-io.c::check_super() as btrfs_check_super()
and use it in btrfs-image for extra verification.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
[BUG]
When there are over 32 (in my example, 35) online CPUs, btrfs-image -c9
will just hang.
[CAUSE]
Btrfs-image has a hard coded limit (32) on how many threads we can use.
For the "-t" option we do the up limit check.
But when we don't specify "-t" option and speicified "-c" option, then
btrfs-image will try to auto detect the number of online CPUs, and use
it without checking if it's over the up limit.
And for num_threads larger than the up limit, we will over write the
adjust members of metadump_struct/mdrestore_struct, corrupting
pthread_mutex_t and pthread_cond_t, causing synchronising problem.
Nowadays, with SMT/HT and higher cpu core counts, it's not hard to go
beyond 32 threads, and hit the bug.
[FIX]
Just do extra num_threads check before using the number from sysconf().
Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba