The new image has a bad data extent item generation:
item 0 key (13631488 EXTENT_ITEM 4096) itemoff 16230 itemsize 53
refs 1 gen 16384 flags DATA
extent data backref root FS_TREE objectid 257 offset 0 count 1
Just like what older `btrfs check --init-extent-tree` could cause.
Reviewed-by: Su Yue <Damenly_Su@gmx.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The manual page of btrfsck clearly states 'btrfs check --repair' is a
dangerous operation.
Although this warning is in place users do not read the manual page
and/or are used to the behaviour of fsck utilities which repair the
filesystem, and thus potentially cause harm.
Similar to 'btrfs balance' without any filters, add a warning and a
countdown, so users can bail out before eventual corrupting the
filesystem more than it already is.
To override the timeout, let --force skip it and continue.
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: David Sterba <dsterba@suse.com>
The image contains one inode item with invalid generation. The image
can be crafted by "btrfs-corrupt-block -i 257 -f generation". It should
emulate the bad inode generation caused by older kernel around 2014.
The image is repairable for both original and lowmem mode.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add new test image for imode repair in subvolume trees.
The new test cases including the following cases:
- Regular file with bad imode
It still has the valid INODE_REF and parent dir has correct DIR_INDEX
and DIR_ITEM.
In this case, no matter if the file is empty or not, it should be
repaired using the info from DIR_INDEX of parent dir.
- Non-empty regular file with bad imode, and without INODE_REF
The file should be mostly an orphan, so no INODE_REF for imode lookup.
But it has EXTENT_DATA which should be enough for imode repair.
The repair also involves moving the orphan to lost+found dir.
- Non-empty dir with bad imode, and without INODE_REF
Pretty much the same case, but now a directory.
The repair also involves moving the orphan to lost+found dir.
Also rename the existing test case 039-bad-free-space-cache-inode-mode
to 039-bad-inode-mode, since now we can fix all bad imode.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Btrfs check used to report false alerts on half dropped orphan inodes.
Add test cases to prevent such problem from happening again.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The image contains a fs tree whose generation is over 100 larger than
super block generation.
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The image has one free space cache inode with invalid mode (0).
item 9 key (256 INODE_ITEM 0) itemoff 13702 itemsize 160
generation 30 transid 30 size 65536 nbytes 1507328
block group 0 mode 0 links 1 uid 0 gid 0 rdev 0
sequence 23 flags 0x1b(NODATASUM|NODATACOW|NOCOMPRESS|PREALLOC)
atime 0.0 (1970-01-01 08:00:00)
ctime 1553491158.189771625 (2019-03-25 13:19:18)
mtime 0.0 (1970-01-01 08:00:00)
otime 0.0 (1970-01-01 08:00:00)
Both lowmem and original mode should be able to detect and fix it.
The extracted test image is pretty big (1G extracted), as kernel won't
cache small chunks.
Even with SSD, such test may still take some seconds just extracting the
image.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Lowmem can repair after commit
'btrfs-progs: lowmem: move nbytes check before isize check',
so add the beacon file.
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
The missing extent will lead to the existence of the gap between adjacent
extents. The fsck should can detect the gap correctly and repair by punch
a hole.
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
The runtime of the test is over 4 minutes when hosted on NFS, the
fallocate phase takes the most time. As fallocate can't take multiple
arguments, we can't save the overhead by creating the files at once.
Create the files in batches in the background, this improves the runtime
to 2 minutes.
Signed-off-by: David Sterba <dsterba@suse.com>
Now two locations can detect such problem, either by device item
used/total bytes check, or by early dev extents check against device
boundary.
The image is hand-crafted image which uses DATA SINGLE chunk to feed
btrfs check. As expected, as long as block group item, chunk item,
device used bytes match, older versions check can't detect such problem.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
For reloc tree, despite of its short lifespan, it's still the backref,
where reloc tree root backref points back to itself, makes it special.
So it's more approriate to put them into 020-extent-ref-cases.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Simple test case which preps a filesystem, then corrupts the FST and
finally repairs it. Tests both extent based and bitmap based FSTs.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The commit d99615284a ("btrfs-progs:
fsck-tests: Add test image to check if btrfs check reports uninitialized
rescan as error") added test 035, should have been 036.
Signed-off-by: David Sterba <dsterba@suse.com>
For trees have been balanced, leaves are with flag BTRFS_HEADER_FLAG_RELOC and
extent data backrefs are shared. Like:
=====================
item 0 key (11927552 EXTENT_ITEM 524288) itemoff 3932 itemsize 63
refs 129 gen 7 flags DATA
shared data backref parent 35897344 count 41
shared data backref parent 35426304 count 37
shared data backref parent 35422208 count 51
=====================
Then make the leaf which owns the extent data cowed. The shared data
backref was to transferred to keyed data ref, but remaining backrefs are
still shared. Like:
=====================
item 0 key (11927552 EXTENT_ITEM 524288) itemoff 3887 itemsize 108
refs 129 gen 7 flags DATA
extent data backref root 5 objectid 258 offset 0 count 40
extent data backref root 5 objectid 257 offset 0 count 1
shared data backref parent 35426304 count 37
shared data backref parent 35422208 count 51
=====================
However lowmem mode used to iterate the whole inode to find all
references, and doesn't care if a reference is already counted by the
shared tree block.
Add the test case to check it.
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The restore target file does not exist and creating by root does not
work on NFS, so precreating will make that work. Also fix the image name
to be deleted.
Signed-off-by: David Sterba <dsterba@suse.com>
There are two bad symlinks in the test case. One is with immutable
attribute. Another one is with append attribute.
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Verify that if we have an otherwise clean filesystem, containging
collided DIR_ITEM, btrfs check lowmem's mode can correctly handle those
and not produce any false positives.
This if fixed by commit titled:
"btrfs-progs: check: fix DIR_ITEM checking in lowmem"
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Since the test case uses run_mustfail(), which is pretty easy pass due to
other unexpected problems, so here an extra run_check() is added to
ensure we don't only report qgroup error, but also fix it without
problem.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
This new image misses one extent which leads lowmem mode to allocate new
chunks in repair.
Rename original image to no_extent_bad_dev.img.
Because of its bad used bytes, it should let lowmem mode
exclude blocks in repair.
Due to problems of btrfs-image, choose xz as compression tool.
Signed-off-by: Su Yue <suy.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Verify that a filesystem check operation (fsck) does not report the
following scenario as an error:
An extent is shared between two inodes, as a result of clone/reflink
operation, and for one of the inodes, lets call it inode A, the extent is
referenced through a file extent item as a prealloc extent, while for the
other inode, call it inode B, the extent is referenced through a regular
file extent item, that is, it was written to. The goal of this test is to
make sure a filesystem check operation will not report "odd csum items"
errors for the prealloc extent at inode A, because this scenario is valid
since the extent was written through inode B and therefore it is expected
to have checksum items in the filesystem's checksum btree for that shared
extent.
Such scenario can be created with the following steps for example:
mkfs.btrfs -f /dev/sdb
mount /dev/sdb /mnt
touch /mnt/foo
xfs_io -c "falloc 0 256K" /mnt/foo
sync
xfs_io -c "pwrite -S 0xab 0 256K" /mnt/foo
touch /mnt/bar
xfs_io -c "reflink /mnt/foo 0 0 256K" /mnt/bar
xfs_io -c "fsync" /mnt/bar
<power fail>
mount /dev/sdb /mnt
umount /mnt
This scenario is fixed by the following patch for the filesystem checker:
"Btrfs-progs: check, fix false error reports for shared prealloc extents"
Signed-off-by: Filipe Manana <fdmanana@suse.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The commit cebf3b3722 ("btrfs-progs: introduce TEST_TOP and
INTERNAL_BIN for tests") did not convert all test paths. This would
break the exported testsutie.
Signed-off-by: David Sterba <dsterba@suse.com>
Regression test for false alerts in lowmem mode.
Signed-off-by: Qu Wenruo <wqu@suse.com>
[ update test ]
Signed-off-by: David Sterba <dsterba@suse.com>
Add the testcase for false alert of data extent backref lost with the
extent offset.
The image can be reproduced by the following commands:
------
dev=~/test.img
mnt=/mnt/btrfs
umount $mnt &> /dev/null
fallocate -l 128M $dev
mkfs.btrfs $dev
mount $dev $mnt
for i in `seq 1 10`; do
xfs_io -f -c "pwrite 0 2K" $mnt/file$i
done
xfs_io -f -c "falloc 0 64K" $mnt/file11
for i in `seq 1 32`; do
xfs_io -f -c "reflink $mnt/file11 0 $(($i * 64))K 64K" $mnt/file11
done
xfs_io -f -c "reflink $mnt/file11 32K $((33 * 64))K 32K" $mnt/file11
btrfs subvolume snapshot $mnt $mnt/snap1
umount $mnt
btrfs-image -c9 $dev extent_data_ref.img
------
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Use TEST_TOP as base for tests to reference any files, this will be used
for git and external testsuite.
INTERNAL_BIN is needed for referencing binaries that could reside in
different paths in git vs external testsuite.
Signed-off-by: Gu Jinxiang <gujx@cn.fujitsu.com>
[ add quotes around sourced files, update changelog ]
Signed-off-by: David Sterba <dsterba@suse.com>
There are 2 fsck tests with the same number 027:
tree-reloc-tree
bad-extent-inline-ref-type
And we also have a hole in 015, so just rename tree-reloc-tree to 015,
to get rid of the duplicated test number and fill in the hole.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Transform the test to the common helpers and don't manage the loop
devices here. The test category changes from check to misc.
Signed-off-by: David Sterba <dsterba@suse.com>
This functionality regressed some time ago and it was never caught. Seems no
one complained of that, but to be sure add a regression test to prevent future
regressions.
Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Reloc tree is a special tree with very short life span. It acts as a
special snapshot for any tree, with related nodes/leaves or EXTENT_DATA
modified to point to new position.
Considering the short life span and its special purpose, it should be
quite reasonable to keep them as both corner case for fsck and
educational dump for anyone interested in relocation.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
For snapshot shared tree blocks with source subvolume, the keyed backref
counter only counts the exclusive owned references.
In the following case, 258 is a snapshot of 257, which inherits all the
reference to this data extent.
------
item 4 key (12582912 EXTENT_ITEM 524288) itemoff 3741 itemsize 140
refs 179 gen 9 flags DATA
extent data backref root 257 objectid 258 offset 0 count 49
extent data backref root 257 objectid 257 offset 0 count 1
extent data backref root 256 objectid 258 offset 0 count 128
extent data backref root 256 objectid 257 offset 0 count 1
------
However lowmem mode used to iterate the whole inode to find all
references, and doesn't care if a reference is already counted by the
shared tree block.
Add the test case to check it.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
The image is dumped by modifying kernel to sleep long enough before
merging relocation trees, so we can just copy the whole image to other
place before kernel begins to merge reloc trees.
And the base image is created by the following script to bump metadata
size:
------
dev=~/test.img
mnt=/mnt/btrfs
umount $mnt &> /dev/null
fallocate -l 128M $dev
mkfs.btrfs -f -n 4k -m single -d single $dev
mount $dev $mnt -o nospace_cache,max_inline=2048
btrfs subvolume create $mnt/src
for i in $(seq -w 0 128); do
xfs_io -f -c "pwrite 0 2k" $mnt/src/file_$i > /dev/null
done
for i in $(seq -w 0 64); do
btrfs subvolume snapshot $mnt/src/ $mnt/snapshot_$i
touch $mnt/snapshot_$i/new
done
sync
------
The image triggers several corner cases that the old lowmem mode didn't
consider.
Like metadata backref with FULL_BACKREF flag and only SHARED_BLOCK_REF
backrefs for metadata. And several tree reloc trees with shared
leaves/nodes to confuse old lowmem mode.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Introduce a new test image, which has an extent item with no inlined
extent data ref, but all keyed extent data ref.
Only in this case we can trigger fase data extent backref lost bug in
lowmem mode.
Signed-off-by: Qu Wenruo <wqu@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Add a image which can reproduce the extent item referencer count
mismatch false alert for lowmem mode.
Reported-by: Marc MERLIN <marc@merlins.org>
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo