[gparted] Use btrfs inspect-internal dump-super to read usage (!105)
- From: Curtis Gedak <gedakc src gnome org>
- To: commits-list gnome org
- Cc:
- Subject: [gparted] Use btrfs inspect-internal dump-super to read usage (!105)
- Date: Thu, 25 Aug 2022 15:55:10 +0000 (UTC)
commit 13c08808ae96e7e00cbec779277ccc313c4fc9b9
Author: Mike Fleetwood <mike fleetwood googlemail com>
Date: Wed Jul 27 23:06:57 2022 +0100
Use btrfs inspect-internal dump-super to read usage (!105)
GParted has been using 'btrfs filesystem show' to report file system
usage but that doesn't work on a file system image so doesn't work in a
GitLab CI test job, as discussed earlier in this patchset.
There is 'btrfs inspect-internal min-dev-size' but:
1. That only works on a mounted file system and GParted isn't going to
mount an unmounted file system just to query it's used space, so by
extension won't work on image files.
2. It reports a figure which is almost the same as the chunk usage of
the device within the btrfs file system. However if some files have
been deleted leaving chunks partially used, then 'btrfs filesystem
resize' will successfully shrink a btrfs smaller than the reported
minimum device size.
And there is also 'btrfs filesystem usage' but that also only works on a
mounted file system.
So instead use 'btrfs inspect-internal dump-super' to report some of the
figures previously obtained from 'btrfs filesystem show'. For example
for a single device btrfs in an image file:
$ truncate -s 256M /tmp/test.img
$ mkfs.btrfs /tmp/test.img
$ btrfs inspect-internal dump-super /tmp/test.img | egrep 'total_bytes|bytes_used|sectorsize|devid'
total_bytes 268435456
bytes_used 114688
sectorsize 4096
dev_item.total_bytes 268435456
dev_item.bytes_used 92274688
dev_item.devid 1
Comparing with results from 'btrfs filesystem show' for the same file
system, after adding a loop device to allow 'btrfs filesystem show' to
succeed:
$ su -
# losetup --find --show /tmp/test.img
# btrfs filesystem show --raw /dev/loop0
Label: none uuid: 32a1eb31-4691-41ae-9ede-c45d723655a3
Total devices 1 FS bytes used 114688
devid 1 size 268435456 used 92274688 path /dev/loop0
This does bring a forced change in the calculation which affects multi-
device btrfs file systems. 'btrfs filesystem show' provided chunk
allocation information per device ("used" figure for each "devid"). The
file system wide used bytes ("FS bytes used") was apportioned according
to the fraction of the chunk allocation each device contained. However
'btrfs inspect-internal dump-super' doesn't provide chunk allocation
information for all devices, only for the current device
("dev_item.bytes_used"). Instead the calculation now has to apportion
the file system wide used bytes ("bytes_used") according to the fraction
of the size of the current device ("dev_item.total_bytes") within the
total size ("total_bytes").
This can't make any difference to a single device btrfs file system as
both fractions will be 1. It only affects how the file system wide used
bytes is distributed among multiple devices.
As an example to see the difference between calculation methods, create
a 2 GiB btrfs taking the defaults so getting duplicated metadata and
single data. Add another 2 GiB partition and populate with some files.
# mkfs.btrfs /dev/sdb1
btrfs-progs v4.15.1
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: 68195e7e-c13f-4095-945f-675af4b1a451
Node size: 16384
Sector size: 4096
Filesystem size: 2.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 102.38MiB
System: DUP 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 2.00GiB /dev/sdb1
# mount /dev/sdb1 /mnt/1
# btrfs device add /dev/sdc1 /mnt/1
# cp -a /home/$USER/programming/c/gparted/ /mnt/1/
Usage figures using the old calculation apportioning file system wide
usage according to chunk allocation per device:
# btrfs filesystem show --raw /dev/sdb1
Label: none uuid: 68195e7e-c13f-4095-945f-675af4b1a451
Total devices 2 FS bytes used 178749440
devid 1 size 2147483648 used 239861760 path /dev/sdb1
devid 2 size 2147483648 used 436207616 path /dev/sdc1
sum_devid_used = 239861760 + 436207616
= 676069376
sdb1 usage = 178749440 * 239861760 / 676069376
= 63418277
sdc1 usage = 178749440 * 436207616 / 676069376
= 115331163
Usage figures using the new calculation apportioning file system wide
usage according to device sizes:
# btrfs inspect-internal dump-super /dev/sdb1 | egrep 'total_bytes|^bytes_used'
total_bytes 4294967296
bytes_used 178749440
dev_item.total_bytes 2147483648
# btrfs inspect-internal dump-super /dev/sdc1 | egrep 'total_bytes|^bytes_used'
total_bytes 4294967296
bytes_used 178749440
dev_item.total_bytes 2147483648
sdb1 usage = 178749440 * 2147483648 / 4294967296
= 89374720
sdc1 usage = 178749440 * 2147483648 / 4294967296
= 89374720
Both calculation methods ignore that btrfs allocates chunks at the
volume manager level. So when fully compacted the last chunk for
metadata and data for each storage profile (RAID level) will be
partially filled and this is not accounted for.
Also for multi-device btrfs file systems the new calculation provides
different results. However given that shrinking a device in a multi-
device btrfs file system can and does relocate extents to other devices
(redundancy requirements of chunks permitting) it's minimum size is
virtually impossible to calculate and may not restrict how small the
btrfs device can be shrunk anyway. If it turns out that this new
calculation causes problems it's been made a separate commit from the
previous commit for easier reverting.
Closes !105 - Update used btrfs file system commands, new minimum is
btrfs-progs 4.5
src/btrfs.cc | 147 +++++++++++++++++++++++++----------------------------------
1 file changed, 61 insertions(+), 86 deletions(-)
---
diff --git a/src/btrfs.cc b/src/btrfs.cc
index 3514ab55..48b4ddb3 100644
--- a/src/btrfs.cc
+++ b/src/btrfs.cc
@@ -156,108 +156,83 @@ void btrfs::set_used_sectors(Partition& partition)
// https://btrfs.wiki.kernel.org/index.php/Glossary
//
// This makes the question of how much disk space is being used in an individual
- // device a complicated question to answer. Further, the current btrfs tools
- // don't provide the required information.
+ // device a complicated question to answer. Additionally, even if there is a
+ // correct answer for the usage / minimum size a device can be, a multi-device
+ // btrfs can and does relocate extents to other devices allowing it to be shrunk
+ // smaller than it's minimum size (redundancy requirements of chunks permitting).
//
- // Btrfs filesystem show only provides space usage information at the chunk level
- // per device. At the file extent level only a single figure for the whole file
- // system is provided. It also reports size of the data and metadata being
- // stored, not the larger figure of the amount of space taken after redundancy is
- // applied. So it is impossible to answer the question of how much disk space is
- // being used in an individual device. Example:
+ // Btrfs inspect-internal dump-super provides chunk allocation information for the
+ // current device only and a single file system wide extent level usage figure.
+ // Calculate the per device used figure as the fraction of file system wide extent
+ // usage apportioned per device.
//
+ // Example:
// # btrfs filesystem show --raw /dev/sdb1
- // Label: none uuid: 003a619e-856f-4b9c-bd29-4d0ae0296d66
- // Total devices 2 FS bytes used 178765824
+ // Label: none uuid: 68195e7e-c13f-4095-945f-675af4b1a451
+ // Total devices 2 FS bytes used 178749440
// devid 1 size 2147483648 used 239861760 path /dev/sdb1
// devid 2 size 2147483648 used 436207616 path /dev/sdc1
//
- // Guesstimate the per device used figure as the fraction of the file system wide
- // extent usage based on chunk usage per device.
+ // # btrfs inspect-internal dump-super /dev/sdb1 | egrep 'total_bytes|bytes_used|sectorsize|devid'
+ // total_bytes 4294967296
+ // bytes_used 178749440
+ // sectorsize 4096
+ // dev_item.total_bytes 2147483648
+ // dev_item.bytes_used 239861760
+ // dev_item.devid 1
//
- // Positives:
- // 1) Per device used figure will correctly be between zero and allocated chunk
- // size.
+ // Calculation:
+ // ptn_fs_size = dev_item_total_bytes
+ // ptn_fs_used = bytes_used * dev_item_total_bytes / total_bytes
//
- // Known inaccuracies:
- // [for single and multi-device btrfs file systems]
- // 1) Btrfs filesystem show reports file system wide file extent usage without
- // considering redundancy applied to that data. (By default btrfs stores two
- // copies of metadata and one copy of data).
- // 2) At minimum size when all data has been consolidated there will be a few
- // partly filled chunks of 256 MiB or more for data and metadata of each
- // storage profile (RAID level).
- // [for multi-device btrfs file systems only]
- // 3) Data may be far from evenly distributed between the chunks on multiple
- // devices.
- // 4) Extents can be and are relocated to other devices within the file system
- // when shrinking a device.
- Utils::execute_command("btrfs filesystem show --raw " + Glib::shell_quote(partition.get_path()),
+ // This calculation also ignores that btrfs allocates chunks at the volume manager
+ // level. So when fully compacted there will be partially filled chunks for
+ // metadata and data for each storage profile (RAID level) not accounted for.
+ Utils::execute_command("btrfs inspect-internal dump-super " + Glib::shell_quote(partition.get_path()),
output, error, true);
- // In many cases the exit status doesn't reflect valid output or an error
- // condition so rely on parsing the output to determine success.
+ // btrfs inspect-internal dump-super returns zero exit status for both success and
+ // failure. Instead use non-empty stderr to identify failure.
+ if (! error.empty())
+ {
+ if (! output.empty())
+ partition.push_back_message(output);
+ if (! error.empty())
+ partition.push_back_message(error);
+ return;
+ }
- // Extract the per device size figure. Guesstimate the per device used
- // figure as discussed above. Example:
- //
- // # btrfs filesystem show --raw /dev/sdb1
- // Label: none uuid: 003a619e-856f-4b9c-bd29-4d0ae0296d66
- // Total devices 2 FS bytes used 178765824
- // devid 1 size 2147483648 used 239861760 path /dev/sdb1
- // devid 2 size 2147483648 used 436207616 path /dev/sdc1
- //
- // Calculations:
- // ptn fs size = devid size
- // ptn fs used = total fs used * devid used / sum devid used
+ // Btrfs file system wide size (sum of devid sizes)
+ long long total_bytes = -1;
+ Glib::ustring::size_type index = output.find("\ntotal_bytes");
+ if (index < output.length())
+ sscanf(output.substr(index).c_str(), "\ntotal_bytes %lld", &total_bytes);
- long long total_fs_used = -1;
- long long sum_devid_used = 0;
- long long devid_used = -1;
- long long devid_size = -1;
+ // Btrfs file system wide used bytes
+ long long bytes_used = -1;
+ index = output.find("\nbytes_used");
+ if (index < output.length())
+ sscanf(output.substr(index).c_str(), "\nbytes_used %lld", &bytes_used);
- // Btrfs file system wide used bytes (extents and items)
- Glib::ustring::size_type index = output.find("FS bytes used");
+ // Sector size
+ long long sector_size = -1;
+ index = output.find("\nsectorsize");
if (index < output.length())
- sscanf(output.substr(index).c_str(), "FS bytes used %lld", &total_fs_used);
+ sscanf(output.substr(index).c_str(), "\nsectorsize %lld", §or_size);
- Glib::ustring::size_type offset = 0 ;
- while ( ( index = output .find( "devid ", offset ) ) != Glib::ustring::npos )
- {
- Glib::ustring devid_path = Utils::regexp_label( output .substr( index ),
- "devid .* path (/dev/[[:graph:]]+)" ) ;
- if ( ! devid_path .empty() )
- {
- // Btrfs per devid used bytes (chunks)
- long long used = -1;
- sscanf(output.substr(index).c_str(), "devid %*d size %*d used %lld path", &used);
- if (used > -1)
- {
- sum_devid_used += used ;
- if ( devid_path == partition .get_path() )
- devid_used = used ;
- }
-
- if ( devid_path == partition .get_path() )
- // Btrfs per device size bytes (chunks)
- sscanf(output.substr(index).c_str(), "devid %*d size %lld used", &devid_size);
- }
- offset = index + 5 ; //Next find starts immediately after current "devid"
- }
+ // Btrfs this device size
+ long long dev_item_total_bytes = -1;
+ index = output.find("\ndev_item.total_bytes");
+ if (index < output.length())
+ sscanf(output.substr(index).c_str(), "\ndev_item.total_bytes %lld", &dev_item_total_bytes);
- if ( total_fs_used > -1 && devid_size > -1 && devid_used > -1 && sum_devid_used > 0 )
+ if (total_bytes > -1 && bytes_used > -1 && dev_item_total_bytes > -1 && sector_size > -1)
{
- T = Utils::round( devid_size / double(partition .sector_size) ) ; //ptn fs size
- double ptn_fs_used = total_fs_used * ( devid_used / double(sum_devid_used) ) ; //ptn fs used
- N = T - Utils::round( ptn_fs_used / double(partition .sector_size) ) ;
- partition .set_sector_usage( T, N ) ;
- }
- else
- {
- if ( ! output .empty() )
- partition.push_back_message( output );
-
- if ( ! error .empty() )
- partition.push_back_message( error );
+ Sector ptn_fs_size = dev_item_total_bytes / partition.sector_size;
+ double devid_size_fraction = dev_item_total_bytes / double(total_bytes);
+ Sector ptn_fs_used = Utils::round(bytes_used * devid_size_fraction) / partition.sector_size;
+ Sector ptn_fs_free = ptn_fs_size - ptn_fs_used;
+ partition.set_sector_usage(ptn_fs_size, ptn_fs_free);
+ partition.fs_block_size = sector_size;
}
}
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
