diff options
Diffstat (limited to 'Documentation/admin-guide/device-mapper')
16 files changed, 2058 insertions, 63 deletions
diff --git a/Documentation/admin-guide/device-mapper/cache-policies.rst b/Documentation/admin-guide/device-mapper/cache-policies.rst index b17fe352fc41..13da4d831d46 100644 --- a/Documentation/admin-guide/device-mapper/cache-policies.rst +++ b/Documentation/admin-guide/device-mapper/cache-policies.rst @@ -70,7 +70,7 @@ the entries (each hotspot block covers a larger area than a single cache block). All this means smq uses ~25bytes per cache block. Still a lot of -memory, but a substantial improvement nontheless. +memory, but a substantial improvement nonetheless. Level balancing ^^^^^^^^^^^^^^^ diff --git a/Documentation/admin-guide/device-mapper/dm-crypt.rst b/Documentation/admin-guide/device-mapper/dm-crypt.rst index 8f4a3f889d43..aa2d04d95df6 100644 --- a/Documentation/admin-guide/device-mapper/dm-crypt.rst +++ b/Documentation/admin-guide/device-mapper/dm-crypt.rst @@ -46,7 +46,7 @@ Parameters:: capi:authenc(hmac(sha256),xts(aes))-random capi:rfc7539(chacha20,poly1305)-random - The /proc/crypto contains a list of curently loaded crypto modes. + The /proc/crypto contains a list of currently loaded crypto modes. <key> Key used for encryption. It is encoded either as a hexadecimal number @@ -67,7 +67,7 @@ Parameters:: the value passed in <key_size>. <key_type> - Either 'logon' or 'user' kernel key type. + Either 'logon', 'user', 'encrypted' or 'trusted' kernel key type. <key_description> The kernel keyring key description crypt target should look for @@ -92,7 +92,7 @@ Parameters:: <#opt_params> Number of optional parameters. If there are no optional parameters, - the optional paramaters section can be skipped or #opt_params can be zero. + the optional parameters section can be skipped or #opt_params can be zero. Otherwise #opt_params is the number of following arguments. Example of optional parameters section: @@ -121,6 +121,14 @@ submit_from_crypt_cpus thread because it benefits CFQ to have writes submitted using the same context. +no_read_workqueue + Bypass dm-crypt internal workqueue and process read requests synchronously. + +no_write_workqueue + Bypass dm-crypt internal workqueue and process write requests synchronously. + This option is automatically enabled for host-managed zoned block devices + (e.g. host-managed SMR hard-disks). + integrity:<bytes>:<type> The device requires additional <bytes> metadata per-sector stored in per-bio integrity structure. This metadata must by provided diff --git a/Documentation/admin-guide/device-mapper/dm-dust.rst b/Documentation/admin-guide/device-mapper/dm-dust.rst index b6e7e7ead831..e35ec8cd2f88 100644 --- a/Documentation/admin-guide/device-mapper/dm-dust.rst +++ b/Documentation/admin-guide/device-mapper/dm-dust.rst @@ -69,10 +69,11 @@ Create the dm-dust device: $ sudo dmsetup create dust1 --table '0 33552384 dust /dev/vdb1 0 4096' Check the status of the read behavior ("bypass" indicates that all I/O -will be passed through to the underlying device):: +will be passed through to the underlying device; "verbose" indicates that +bad block additions, removals, and remaps will be verbosely logged):: $ sudo dmsetup status dust1 - 0 33552384 dust 252:17 bypass + 0 33552384 dust 252:17 bypass verbose $ sudo dd if=/dev/mapper/dust1 of=/dev/null bs=512 count=128 iflag=direct 128+0 records in @@ -164,7 +165,7 @@ following message command:: A message will print with the number of bad blocks currently configured on the device:: - kernel: device-mapper: dust: countbadblocks: 895 badblock(s) found + countbadblocks: 895 badblock(s) found Querying for specific bad blocks -------------------------------- @@ -176,11 +177,11 @@ following message command:: The following message will print if the block is in the list:: - device-mapper: dust: queryblock: block 72 found in badblocklist + dust_query_block: block 72 found in badblocklist The following message will print if the block is not in the list:: - device-mapper: dust: queryblock: block 72 not found in badblocklist + dust_query_block: block 72 not found in badblocklist The "queryblock" message command will work in both the "enabled" and "disabled" modes, allowing the verification of whether a block @@ -198,12 +199,28 @@ following message command:: After clearing the bad block list, the following message will appear:: - kernel: device-mapper: dust: clearbadblocks: badblocks cleared + dust_clear_badblocks: badblocks cleared If there were no bad blocks to clear, the following message will appear:: - kernel: device-mapper: dust: clearbadblocks: no badblocks found + dust_clear_badblocks: no badblocks found + +Listing the bad block list +-------------------------- + +To list all bad blocks in the bad block list (using an example device +with blocks 1 and 2 in the bad block list), run the following message +command:: + + $ sudo dmsetup message dust1 0 listbadblocks + 1 + 2 + +If there are no bad blocks in the bad block list, the command will +execute with no output:: + + $ sudo dmsetup message dust1 0 listbadblocks Message commands list --------------------- @@ -223,6 +240,7 @@ Single argument message commands:: countbadblocks clearbadblocks + listbadblocks disable enable quiet diff --git a/Documentation/admin-guide/device-mapper/dm-ebs.rst b/Documentation/admin-guide/device-mapper/dm-ebs.rst new file mode 100644 index 000000000000..c09f66db5621 --- /dev/null +++ b/Documentation/admin-guide/device-mapper/dm-ebs.rst @@ -0,0 +1,51 @@ +====== +dm-ebs +====== + + +This target is similar to the linear target except that it emulates +a smaller logical block size on a device with a larger logical block +size. Its main purpose is to provide emulation of 512 byte sectors on +devices that do not provide this emulation (i.e. 4K native disks). + +Supported emulated logical block sizes 512, 1024, 2048 and 4096. + +Underlying block size can be set to > 4K to test buffering larger units. + + +Table parameters +---------------- + <dev path> <offset> <emulated sectors> [<underlying sectors>] + +Mandatory parameters: + + <dev path>: + Full pathname to the underlying block-device, + or a "major:minor" device-number. + <offset>: + Starting sector within the device; + has to be a multiple of <emulated sectors>. + <emulated sectors>: + Number of sectors defining the logical block size to be emulated; + 1, 2, 4, 8 sectors of 512 bytes supported. + +Optional parameter: + + <underlying sectors>: + Number of sectors defining the logical block size of <dev path>. + 2^N supported, e.g. 8 = emulate 8 sectors of 512 bytes = 4KiB. + If not provided, the logical block size of <dev path> will be used. + + +Examples: + +Emulate 1 sector = 512 bytes logical block size on /dev/sda starting at +offset 1024 sectors with underlying devices block size automatically set: + +ebs /dev/sda 1024 1 + +Emulate 2 sector = 1KiB logical block size on /dev/sda starting at +offset 128 sectors, enforce 2KiB underlying device block size. +This presumes 2KiB logical blocksize on /dev/sda or less to work: + +ebs /dev/sda 128 2 4 diff --git a/Documentation/admin-guide/device-mapper/dm-flakey.rst b/Documentation/admin-guide/device-mapper/dm-flakey.rst index 86138735879d..f967c5fea219 100644 --- a/Documentation/admin-guide/device-mapper/dm-flakey.rst +++ b/Documentation/admin-guide/device-mapper/dm-flakey.rst @@ -39,6 +39,10 @@ Optional feature parameters: If no feature parameters are present, during the periods of unreliability, all I/O returns errors. + error_reads: + All read I/O is failed with an error signalled. + Write I/O is handled correctly. + drop_writes: All write I/O is silently ignored. Read I/O is handled correctly. @@ -63,6 +67,16 @@ Optional feature parameters: Perform the replacement only if bio->bi_opf has all the selected flags set. + random_read_corrupt <probability> + During <down interval>, replace random byte in a read bio + with a random value. probability is an integer between + 0 and 1000000000 meaning 0% to 100% probability of corruption. + + random_write_corrupt <probability> + During <down interval>, replace random byte in a write bio + with a random value. probability is an integer between + 0 and 1000000000 meaning 0% to 100% probability of corruption. + Examples: Replaces the 32nd byte of READ bios with the value 1:: diff --git a/Documentation/admin-guide/device-mapper/dm-ima.rst b/Documentation/admin-guide/device-mapper/dm-ima.rst new file mode 100644 index 000000000000..a4aa50a828e0 --- /dev/null +++ b/Documentation/admin-guide/device-mapper/dm-ima.rst @@ -0,0 +1,715 @@ +====== +dm-ima +====== + +For a given system, various external services/infrastructure tools +(including the attestation service) interact with it - both during the +setup and during rest of the system run-time. They share sensitive data +and/or execute critical workload on that system. The external services +may want to verify the current run-time state of the relevant kernel +subsystems before fully trusting the system with business-critical +data/workload. + +Device mapper plays a critical role on a given system by providing +various important functionalities to the block devices using various +target types like crypt, verity, integrity etc. Each of these target +types’ functionalities can be configured with various attributes. +The attributes chosen to configure these target types can significantly +impact the security profile of the block device, and in-turn, of the +system itself. For instance, the type of encryption algorithm and the +key size determines the strength of encryption for a given block device. + +Therefore, verifying the current state of various block devices as well +as their various target attributes is crucial for external services before +fully trusting the system with business-critical data/workload. + +IMA kernel subsystem provides the necessary functionality for +device mapper to measure the state and configuration of +various block devices - + +- by device mapper itself, from within the kernel, +- in a tamper resistant way, +- and re-measured - triggered on state/configuration change. + +Setting the IMA Policy: +======================= +For IMA to measure the data on a given system, the IMA policy on the +system needs to be updated to have following line, and the system needs +to be restarted for the measurements to take effect. + +:: + + /etc/ima/ima-policy + measure func=CRITICAL_DATA label=device-mapper template=ima-buf + +The measurements will be reflected in the IMA logs, which are located at: + +:: + + /sys/kernel/security/integrity/ima/ascii_runtime_measurements + /sys/kernel/security/integrity/ima/binary_runtime_measurements + +Then IMA ASCII measurement log has the following format: + +:: + + <PCR> <TEMPLATE_DATA_DIGEST> <TEMPLATE_NAME> <TEMPLATE_DATA> + + PCR := Platform Configuration Register, in which the values are registered. + This is applicable if TPM chip is in use. + + TEMPLATE_DATA_DIGEST := Template data digest of the IMA record. + TEMPLATE_NAME := Template name that registered the integrity value (e.g. ima-buf). + + TEMPLATE_DATA := <ALG> ":" <EVENT_DIGEST> <EVENT_NAME> <EVENT_DATA> + It contains data for the specific event to be measured, + in a given template data format. + + ALG := Algorithm to compute event digest + EVENT_DIGEST := Digest of the event data + EVENT_NAME := Description of the event (e.g. 'dm_table_load'). + EVENT_DATA := The event data to be measured. + +| + +| *NOTE #1:* +| The DM target data measured by IMA subsystem can alternatively + be queried from userspace by setting DM_IMA_MEASUREMENT_FLAG with + DM_TABLE_STATUS_CMD. + +| + +| *NOTE #2:* +| The Kernel configuration CONFIG_IMA_DISABLE_HTABLE allows measurement of duplicate records. +| To support recording duplicate IMA events in the IMA log, the Kernel needs to be configured with + CONFIG_IMA_DISABLE_HTABLE=y. + +Supported Device States: +======================== +Following device state changes will trigger IMA measurements: + + 1. Table load + #. Device resume + #. Device remove + #. Table clear + #. Device rename + +1. Table load: +--------------- +When a new table is loaded in a device's inactive table slot, +the device information and target specific details from the +targets in the table are measured. + +The IMA measurement log has the following format for 'dm_table_load': + +:: + + EVENT_NAME := "dm_table_load" + EVENT_DATA := <dm_version_str> ";" <device_metadata> ";" <table_load_data> + + dm_version_str := "dm_version=" <N> "." <N> "." <N> + Same as Device Mapper driver version. + device_metadata := <device_name> "," <device_uuid> "," <device_major> "," <device_minor> "," + <minor_count> "," <num_device_targets> ";" + + device_name := "name=" <dm-device-name> + device_uuid := "uuid=" <dm-device-uuid> + device_major := "major=" <N> + device_minor := "minor=" <N> + minor_count := "minor_count=" <N> + num_device_targets := "num_targets=" <N> + dm-device-name := Name of the device. If it contains special characters like '\', ',', ';', + they are prefixed with '\'. + dm-device-uuid := UUID of the device. If it contains special characters like '\', ',', ';', + they are prefixed with '\'. + + table_load_data := <target_data> + Represents the data (as name=value pairs) from various targets in the table, + which is being loaded into the DM device's inactive table slot. + target_data := <target_data_row> | <target_data><target_data_row> + + target_data_row := <target_index> "," <target_begin> "," <target_len> "," <target_name> "," + <target_version> "," <target_attributes> ";" + target_index := "target_index=" <N> + Represents nth target in the table (from 0 to N-1 targets specified in <num_device_targets>) + If all the data for N targets doesn't fit in the given buffer - then the data that fits + in the buffer (say from target 0 to x) is measured in a given IMA event. + The remaining data from targets x+1 to N-1 is measured in the subsequent IMA events, + with the same format as that of 'dm_table_load' + i.e. <dm_version_str> ";" <device_metadata> ";" <table_load_data>. + + target_begin := "target_begin=" <N> + target_len := "target_len=" <N> + target_name := Name of the target. 'linear', 'crypt', 'integrity' etc. + The targets that are supported for IMA measurements are documented below in the + 'Supported targets' section. + target_version := "target_version=" <N> "." <N> "." <N> + target_attributes := Data containing comma separated list of name=value pairs of target specific attributes. + + For instance, if a linear device is created with the following table entries, + # dmsetup create linear1 + 0 2 linear /dev/loop0 512 + 2 2 linear /dev/loop0 512 + 4 2 linear /dev/loop0 512 + 6 2 linear /dev/loop0 512 + + Then IMA ASCII measurement log will have the following entry: + (converted from ASCII to text for readability) + + 10 a8c5ff755561c7a28146389d1514c318592af49a ima-buf sha256:4d73481ecce5eadba8ab084640d85bb9ca899af4d0a122989252a76efadc5b72 + dm_table_load + dm_version=4.45.0; + name=linear1,uuid=,major=253,minor=0,minor_count=1,num_targets=4; + target_index=0,target_begin=0,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + target_index=1,target_begin=2,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + target_index=2,target_begin=4,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + target_index=3,target_begin=6,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + +2. Device resume: +------------------ +When a suspended device is resumed, the device information and the hash of the +data from previous load of an active table are measured. + +The IMA measurement log has the following format for 'dm_device_resume': + +:: + + EVENT_NAME := "dm_device_resume" + EVENT_DATA := <dm_version_str> ";" <device_metadata> ";" <active_table_hash> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_metadata := As described in the 'Table load' section above. + active_table_hash := "active_table_hash=" <table_hash_alg> ":" <table_hash> + Rerpresents the hash of the IMA data being measured for the + active table for the device. + table_hash_alg := Algorithm used to compute the hash. + table_hash := Hash of the (<dm_version_str> ";" <device_metadata> ";" <table_load_data> ";") + as described in the 'dm_table_load' above. + Note: If the table_load data spans across multiple IMA 'dm_table_load' + events for a given device, the hash is computed combining all the event data + i.e. (<dm_version_str> ";" <device_metadata> ";" <table_load_data> ";") + across all those events. + current_device_capacity := "current_device_capacity=" <N> + + For instance, if a linear device is resumed with the following command, + #dmsetup resume linear1 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 56c00cc062ffc24ccd9ac2d67d194af3282b934e ima-buf sha256:e7d12c03b958b4e0e53e7363a06376be88d98a1ac191fdbd3baf5e4b77f329b6 + dm_device_resume + dm_version=4.45.0; + name=linear1,uuid=,major=253,minor=0,minor_count=1,num_targets=4; + active_table_hash=sha256:4d73481ecce5eadba8ab084640d85bb9ca899af4d0a122989252a76efadc5b72;current_device_capacity=8; + +3. Device remove: +------------------ +When a device is removed, the device information and a sha256 hash of the +data from an active and inactive table are measured. + +The IMA measurement log has the following format for 'dm_device_remove': + +:: + + EVENT_NAME := "dm_device_remove" + EVENT_DATA := <dm_version_str> ";" <device_active_metadata> ";" <device_inactive_metadata> ";" + <active_table_hash> "," <inactive_table_hash> "," <remove_all> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_active_metadata := Device metadata that reflects the currently loaded active table. + The format is same as 'device_metadata' described in the 'Table load' section above. + device_inactive_metadata := Device metadata that reflects the inactive table. + The format is same as 'device_metadata' described in the 'Table load' section above. + active_table_hash := Hash of the currently loaded active table. + The format is same as 'active_table_hash' described in the 'Device resume' section above. + inactive_table_hash := Hash of the inactive table. + The format is same as 'active_table_hash' described in the 'Device resume' section above. + remove_all := "remove_all=" <yes_no> + yes_no := "y" | "n" + current_device_capacity := "current_device_capacity=" <N> + + For instance, if a linear device is removed with the following command, + #dmsetup remove l1 + + then IMA ASCII measurement log will have the following entry: + (converted from ASCII to text for readability) + + 10 790e830a3a7a31590824ac0642b3b31c2d0e8b38 ima-buf sha256:ab9f3c959367a8f5d4403d6ce9c3627dadfa8f9f0e7ec7899299782388de3840 + dm_device_remove + dm_version=4.45.0; + device_active_metadata=name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=2; + device_inactive_metadata=name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=1; + active_table_hash=sha256:4a7e62efaebfc86af755831998b7db6f59b60d23c9534fb16a4455907957953a, + inactive_table_hash=sha256:9d79c175bc2302d55a183e8f50ad4bafd60f7692fd6249e5fd213e2464384b86,remove_all=n; + current_device_capacity=2048; + +4. Table clear: +---------------- +When an inactive table is cleared from the device, the device information and a sha256 hash of the +data from an inactive table are measured. + +The IMA measurement log has the following format for 'dm_table_clear': + +:: + + EVENT_NAME := "dm_table_clear" + EVENT_DATA := <dm_version_str> ";" <device_inactive_metadata> ";" <inactive_table_hash> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_inactive_metadata := Device metadata that was captured during the load time inactive table being cleared. + The format is same as 'device_metadata' described in the 'Table load' section above. + inactive_table_hash := Hash of the inactive table being cleared from the device. + The format is same as 'active_table_hash' described in the 'Device resume' section above. + current_device_capacity := "current_device_capacity=" <N> + + For instance, if a linear device's inactive table is cleared, + #dmsetup clear l1 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 77d347408f557f68f0041acb0072946bb2367fe5 ima-buf sha256:42f9ca22163fdfa548e6229dece2959bc5ce295c681644240035827ada0e1db5 + dm_table_clear + dm_version=4.45.0; + name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=1; + inactive_table_hash=sha256:75c0dc347063bf474d28a9907037eba060bfe39d8847fc0646d75e149045d545;current_device_capacity=1024; + +5. Device rename: +------------------ +When an device's NAME or UUID is changed, the device information and the new NAME and UUID +are measured. + +The IMA measurement log has the following format for 'dm_device_rename': + +:: + + EVENT_NAME := "dm_device_rename" + EVENT_DATA := <dm_version_str> ";" <device_active_metadata> ";" <new_device_name> "," <new_device_uuid> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_active_metadata := Device metadata that reflects the currently loaded active table. + The format is same as 'device_metadata' described in the 'Table load' section above. + new_device_name := "new_name=" <dm-device-name> + dm-device-name := Same as <dm-device-name> described in 'Table load' section above + new_device_uuid := "new_uuid=" <dm-device-uuid> + dm-device-uuid := Same as <dm-device-uuid> described in 'Table load' section above + current_device_capacity := "current_device_capacity=" <N> + + E.g 1: if a linear device's name is changed with the following command, + #dmsetup rename linear1 --setuuid 1234-5678 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 8b0423209b4c66ac1523f4c9848c9b51ee332f48 ima-buf sha256:6847b7258134189531db593e9230b257c84f04038b5a18fd2e1473860e0569ac + dm_device_rename + dm_version=4.45.0; + name=linear1,uuid=,major=253,minor=2,minor_count=1,num_targets=1;new_name=linear1,new_uuid=1234-5678; + current_device_capacity=1024; + + E.g 2: if a linear device's name is changed with the following command, + # dmsetup rename linear1 linear=2 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 bef70476b99c2bdf7136fae033aa8627da1bf76f ima-buf sha256:8c6f9f53b9ef9dc8f92a2f2cca8910e622543d0f0d37d484870cb16b95111402 + dm_device_rename + dm_version=4.45.0; + name=linear1,uuid=1234-5678,major=253,minor=2,minor_count=1,num_targets=1; + new_name=linear\=2,new_uuid=1234-5678; + current_device_capacity=1024; + +Supported targets: +================== + +Following targets are supported to measure their data using IMA: + + 1. cache + #. crypt + #. integrity + #. linear + #. mirror + #. multipath + #. raid + #. snapshot + #. striped + #. verity + +1. cache +--------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'cache' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <metadata_mode> "," <cache_metadata_device> "," + <cache_device> "," <cache_origin_device> "," <writethrough> "," <writeback> "," + <passthrough> "," <no_discard_passdown> ";" + + target_name := "target_name=cache" + target_version := "target_version=" <N> "." <N> "." <N> + metadata_mode := "metadata_mode=" <cache_metadata_mode> + cache_metadata_mode := "fail" | "ro" | "rw" + cache_device := "cache_device=" <cache_device_name_string> + cache_origin_device := "cache_origin_device=" <cache_origin_device_string> + writethrough := "writethrough=" <yes_no> + writeback := "writeback=" <yes_no> + passthrough := "passthrough=" <yes_no> + no_discard_passdown := "no_discard_passdown=" <yes_no> + yes_no := "y" | "n" + + E.g. + When a 'cache' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'cache' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0;name=cache1,uuid=cache_uuid,major=253,minor=2,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=28672,target_name=cache,target_version=2.2.0,metadata_mode=rw, + cache_metadata_device=253:4,cache_device=253:3,cache_origin_device=253:5,writethrough=y,writeback=n, + passthrough=n,metadata2=y,no_discard_passdown=n; + + +2. crypt +--------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'crypt' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <allow_discards> "," <same_cpu_crypt> "," + <submit_from_crypt_cpus> "," <no_read_workqueue> "," <no_write_workqueue> "," + <iv_large_sectors> "," <iv_large_sectors> "," [<integrity_tag_size> ","] [<cipher_auth> ","] + [<sector_size> ","] [<cipher_string> ","] <key_size> "," <key_parts> "," + <key_extra_size> "," <key_mac_size> ";" + + target_name := "target_name=crypt" + target_version := "target_version=" <N> "." <N> "." <N> + allow_discards := "allow_discards=" <yes_no> + same_cpu_crypt := "same_cpu_crypt=" <yes_no> + submit_from_crypt_cpus := "submit_from_crypt_cpus=" <yes_no> + no_read_workqueue := "no_read_workqueue=" <yes_no> + no_write_workqueue := "no_write_workqueue=" <yes_no> + iv_large_sectors := "iv_large_sectors=" <yes_no> + integrity_tag_size := "integrity_tag_size=" <N> + cipher_auth := "cipher_auth=" <string> + sector_size := "sector_size=" <N> + cipher_string := "cipher_string=" + key_size := "key_size=" <N> + key_parts := "key_parts=" <N> + key_extra_size := "key_extra_size=" <N> + key_mac_size := "key_mac_size=" <N> + yes_no := "y" | "n" + + E.g. + When a 'crypt' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'crypt' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=crypt1,uuid=crypt_uuid1,major=253,minor=0,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=1953125,target_name=crypt,target_version=1.23.0, + allow_discards=y,same_cpu=n,submit_from_crypt_cpus=n,no_read_workqueue=n,no_write_workqueue=n, + iv_large_sectors=n,cipher_string=aes-xts-plain64,key_size=32,key_parts=1,key_extra_size=0,key_mac_size=0; + +3. integrity +------------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'integrity' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <dev_name> "," <start> + <tag_size> "," <mode> "," [<meta_device> ","] [<block_size> ","] <recalculate> "," + <allow_discards> "," <fix_padding> "," <fix_hmac> "," <legacy_recalculate> "," + <journal_sectors> "," <interleave_sectors> "," <buffer_sectors> ";" + + target_name := "target_name=integrity" + target_version := "target_version=" <N> "." <N> "." <N> + dev_name := "dev_name=" <device_name_str> + start := "start=" <N> + tag_size := "tag_size=" <N> + mode := "mode=" <integrity_mode_str> + integrity_mode_str := "J" | "B" | "D" | "R" + meta_device := "meta_device=" <meta_device_str> + block_size := "block_size=" <N> + recalculate := "recalculate=" <yes_no> + allow_discards := "allow_discards=" <yes_no> + fix_padding := "fix_padding=" <yes_no> + fix_hmac := "fix_hmac=" <yes_no> + legacy_recalculate := "legacy_recalculate=" <yes_no> + journal_sectors := "journal_sectors=" <N> + interleave_sectors := "interleave_sectors=" <N> + buffer_sectors := "buffer_sectors=" <N> + yes_no := "y" | "n" + + E.g. + When a 'integrity' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'integrity' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=integrity1,uuid=,major=253,minor=1,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=7856,target_name=integrity,target_version=1.10.0, + dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n,fix_padding=n, + fix_hmac=n,legacy_recalculate=n,journal_sectors=88,interleave_sectors=32768,buffer_sectors=128; + + +4. linear +---------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'linear' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <device_name> <,> <start> ";" + + target_name := "target_name=linear" + target_version := "target_version=" <N> "." <N> "." <N> + device_name := "device_name=" <linear_device_name_str> + start := "start=" <N> + + E.g. + When a 'linear' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'linear' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=linear1,uuid=linear_uuid1,major=253,minor=2,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=28672,target_name=linear,target_version=1.4.0, + device_name=253:1,start=2048; + +5. mirror +---------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'mirror' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <nr_mirrors> "," + <mirror_device_data> "," <handle_errors> "," <keep_log> "," <log_type_status> ";" + + target_name := "target_name=mirror" + target_version := "target_version=" <N> "." <N> "." <N> + nr_mirrors := "nr_mirrors=" <NR> + mirror_device_data := <mirror_device_row> | <mirror_device_data><mirror_device_row> + mirror_device_row is repeated <NR> times - for <NR> described in <nr_mirrors>. + mirror_device_row := <mirror_device_name> "," <mirror_device_status> + mirror_device_name := "mirror_device_" <X> "=" <mirror_device_name_str> + where <X> ranges from 0 to (<NR> -1) - for <NR> described in <nr_mirrors>. + mirror_device_status := "mirror_device_" <X> "_status=" <mirror_device_status_char> + where <X> ranges from 0 to (<NR> -1) - for <NR> described in <nr_mirrors>. + mirror_device_status_char := "A" | "F" | "D" | "S" | "R" | "U" + handle_errors := "handle_errors=" <yes_no> + keep_log := "keep_log=" <yes_no> + log_type_status := "log_type_status=" <log_type_status_str> + yes_no := "y" | "n" + + E.g. + When a 'mirror' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'mirror' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=mirror1,uuid=mirror_uuid1,major=253,minor=6,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=2048,target_name=mirror,target_version=1.14.0,nr_mirrors=2, + mirror_device_0=253:4,mirror_device_0_status=A, + mirror_device_1=253:5,mirror_device_1_status=A, + handle_errors=y,keep_log=n,log_type_status=; + +6. multipath +------------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'multipath' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <nr_priority_groups> + ["," <pg_state> "," <priority_groups> "," <priority_group_paths>] ";" + + target_name := "target_name=multipath" + target_version := "target_version=" <N> "." <N> "." <N> + nr_priority_groups := "nr_priority_groups=" <NPG> + priority_groups := <priority_groups_row>|<priority_groups_row><priority_groups> + priority_groups_row := "pg_state_" <X> "=" <pg_state_str> "," "nr_pgpaths_" <X> "=" <NPGP> "," + "path_selector_name_" <X> "=" <string> "," <priority_group_paths> + where <X> ranges from 0 to (<NPG> -1) - for <NPG> described in <nr_priority_groups>. + pg_state_str := "E" | "A" | "D" + <priority_group_paths> := <priority_group_paths_row> | <priority_group_paths_row><priority_group_paths> + priority_group_paths_row := "path_name_" <X> "_" <Y> "=" <string> "," "is_active_" <X> "_" <Y> "=" <is_active_str> + "fail_count_" <X> "_" <Y> "=" <N> "," "path_selector_status_" <X> "_" <Y> "=" <path_selector_status_str> + where <X> ranges from 0 to (<NPG> -1) - for <NPG> described in <nr_priority_groups>, + and <Y> ranges from 0 to (<NPGP> -1) - for <NPGP> described in <priority_groups_row>. + is_active_str := "A" | "F" + + E.g. + When a 'multipath' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'multipath' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=mp,uuid=,major=253,minor=0,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=2097152,target_name=multipath,target_version=1.14.0,nr_priority_groups=2, + pg_state_0=E,nr_pgpaths_0=2,path_selector_name_0=queue-length, + path_name_0_0=8:16,is_active_0_0=A,fail_count_0_0=0,path_selector_status_0_0=, + path_name_0_1=8:32,is_active_0_1=A,fail_count_0_1=0,path_selector_status_0_1=, + pg_state_1=E,nr_pgpaths_1=2,path_selector_name_1=queue-length, + path_name_1_0=8:48,is_active_1_0=A,fail_count_1_0=0,path_selector_status_1_0=, + path_name_1_1=8:64,is_active_1_1=A,fail_count_1_1=0,path_selector_status_1_1=; + +7. raid +-------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'raid' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <raid_type> "," <raid_disks> "," <raid_state> + <raid_device_status> ["," journal_dev_mode] ";" + + target_name := "target_name=raid" + target_version := "target_version=" <N> "." <N> "." <N> + raid_type := "raid_type=" <raid_type_str> + raid_disks := "raid_disks=" <NRD> + raid_state := "raid_state=" <raid_state_str> + raid_state_str := "frozen" | "reshape" |"resync" | "check" | "repair" | "recover" | "idle" |"undef" + raid_device_status := <raid_device_status_row> | <raid_device_status_row><raid_device_status> + <raid_device_status_row> is repeated <NRD> times - for <NRD> described in <raid_disks>. + raid_device_status_row := "raid_device_" <X> "_status=" <raid_device_status_str> + where <X> ranges from 0 to (<NRD> -1) - for <NRD> described in <raid_disks>. + raid_device_status_str := "A" | "D" | "a" | "-" + journal_dev_mode := "journal_dev_mode=" <journal_dev_mode_str> + journal_dev_mode_str := "writethrough" | "writeback" | "invalid" + + E.g. + When a 'raid' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'raid' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=raid_LV1,uuid=uuid_raid_LV1,major=253,minor=12,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=2048,target_name=raid,target_version=1.15.1, + raid_type=raid10,raid_disks=4,raid_state=idle, + raid_device_0_status=A, + raid_device_1_status=A, + raid_device_2_status=A, + raid_device_3_status=A; + + +8. snapshot +------------ +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'snapshot' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <snap_origin_name> "," + <snap_cow_name> "," <snap_valid> "," <snap_merge_failed> "," <snapshot_overflowed> ";" + + target_name := "target_name=snapshot" + target_version := "target_version=" <N> "." <N> "." <N> + snap_origin_name := "snap_origin_name=" <string> + snap_cow_name := "snap_cow_name=" <string> + snap_valid := "snap_valid=" <yes_no> + snap_merge_failed := "snap_merge_failed=" <yes_no> + snapshot_overflowed := "snapshot_overflowed=" <yes_no> + yes_no := "y" | "n" + + E.g. + When a 'snapshot' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'snapshot' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=snap1,uuid=snap_uuid1,major=253,minor=13,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=4096,target_name=snapshot,target_version=1.16.0, + snap_origin_name=253:11,snap_cow_name=253:12,snap_valid=y,snap_merge_failed=n,snapshot_overflowed=n; + +9. striped +----------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'striped' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <stripes> "," <chunk_size> "," + <stripe_data> ";" + + target_name := "target_name=striped" + target_version := "target_version=" <N> "." <N> "." <N> + stripes := "stripes=" <NS> + chunk_size := "chunk_size=" <N> + stripe_data := <stripe_data_row>|<stripe_data><stripe_data_row> + stripe_data_row := <stripe_device_name> "," <stripe_physical_start> "," <stripe_status> + stripe_device_name := "stripe_" <X> "_device_name=" <stripe_device_name_str> + where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>. + stripe_physical_start := "stripe_" <X> "_physical_start=" <N> + where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>. + stripe_status := "stripe_" <X> "_status=" <stripe_status_str> + where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>. + stripe_status_str := "D" | "A" + + E.g. + When a 'striped' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'striped' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=striped1,uuid=striped_uuid1,major=253,minor=5,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=640,target_name=striped,target_version=1.6.0,stripes=2,chunk_size=64, + stripe_0_device_name=253:0,stripe_0_physical_start=2048,stripe_0_status=A, + stripe_1_device_name=253:3,stripe_1_physical_start=2048,stripe_1_status=A; + +10. verity +---------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'verity' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <hash_failed> "," <verity_version> "," + <data_device_name> "," <hash_device_name> "," <verity_algorithm> "," <root_digest> "," + <salt> "," <ignore_zero_blocks> "," <check_at_most_once> ["," <root_hash_sig_key_desc>] + ["," <verity_mode>] ";" + + target_name := "target_name=verity" + target_version := "target_version=" <N> "." <N> "." <N> + hash_failed := "hash_failed=" <hash_failed_str> + hash_failed_str := "C" | "V" + verity_version := "verity_version=" <verity_version_str> + data_device_name := "data_device_name=" <data_device_name_str> + hash_device_name := "hash_device_name=" <hash_device_name_str> + verity_algorithm := "verity_algorithm=" <verity_algorithm_str> + root_digest := "root_digest=" <root_digest_str> + salt := "salt=" <salt_str> + salt_str := "-" <verity_salt_str> + ignore_zero_blocks := "ignore_zero_blocks=" <yes_no> + check_at_most_once := "check_at_most_once=" <yes_no> + root_hash_sig_key_desc := "root_hash_sig_key_desc=" + verity_mode := "verity_mode=" <verity_mode_str> + verity_mode_str := "ignore_corruption" | "restart_on_corruption" | "panic_on_corruption" | "invalid" + yes_no := "y" | "n" + + E.g. + When a 'verity' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'verity' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=test-verity,uuid=,major=253,minor=2,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=1953120,target_name=verity,target_version=1.8.0,hash_failed=V, + verity_version=1,data_device_name=253:1,hash_device_name=253:0,verity_algorithm=sha256, + root_digest=29cb87e60ce7b12b443ba6008266f3e41e93e403d7f298f8e3f316b29ff89c5e, + salt=e48da609055204e89ae53b655ca2216dd983cf3cb829f34f63a297d106d53e2d, + ignore_zero_blocks=n,check_at_most_once=n; diff --git a/Documentation/admin-guide/device-mapper/dm-init.rst b/Documentation/admin-guide/device-mapper/dm-init.rst index e5242ff17e9b..981d6a907699 100644 --- a/Documentation/admin-guide/device-mapper/dm-init.rst +++ b/Documentation/admin-guide/device-mapper/dm-init.rst @@ -123,3 +123,11 @@ Other examples (per target): 0 1638400 verity 1 8:1 8:2 4096 4096 204800 1 sha256 fb1a5a0f00deb908d8b53cb270858975e76cf64105d412ce764225d53b8f3cfd 51934789604d1b92399c52e7cb149d1b3a1b74bbbcb103b2a0aaacbed5c08584 + +For setups using device-mapper on top of asynchronously probed block +devices (MMC, USB, ..), it may be necessary to tell dm-init to +explicitly wait for them to become available before setting up the +device-mapper tables. This can be done with the "dm-mod.waitfor=" +module parameter, which takes a list of devices to wait for:: + + dm-mod.waitfor=<device1>[,..,<deviceN>] diff --git a/Documentation/admin-guide/device-mapper/dm-integrity.rst b/Documentation/admin-guide/device-mapper/dm-integrity.rst index 594095b54b29..d8a5f14d0e3c 100644 --- a/Documentation/admin-guide/device-mapper/dm-integrity.rst +++ b/Documentation/admin-guide/device-mapper/dm-integrity.rst @@ -25,7 +25,7 @@ mode it calculates and verifies the integrity tag internally. In this mode, the dm-integrity target can be used to detect silent data corruption on the disk or in the I/O path. -There's an alternate mode of operation where dm-integrity uses bitmap +There's an alternate mode of operation where dm-integrity uses a bitmap instead of a journal. If a bit in the bitmap is 1, the corresponding region's data and integrity tags are not synchronized - if the machine crashes, the unsynchronized regions will be recalculated. The bitmap mode @@ -38,6 +38,15 @@ the device. But it will only format the device if the superblock contains zeroes. If the superblock is neither valid nor zeroed, the dm-integrity target can't be loaded. +Accesses to the on-disk metadata area containing checksums (aka tags) are +buffered using dm-bufio. When an access to any given metadata area +occurs, each unique metadata area gets its own buffer(s). The buffer size +is capped at the size of the metadata area, but may be smaller, thereby +requiring multiple buffers to represent the full metadata area. A smaller +buffer size will produce a smaller resulting read/write operation to the +metadata area for small reads/writes. The metadata is still read even in +a full write to the data covered by a single buffer. + To use the target for the first time: 1. overwrite the superblock with zeroes @@ -45,7 +54,7 @@ To use the target for the first time: will format the device 3. unload the dm-integrity target 4. read the "provided_data_sectors" value from the superblock -5. load the dm-integrity target with the the target size +5. load the dm-integrity target with the target size "provided_data_sectors" 6. if you want to use dm-integrity with dm-crypt, load the dm-crypt target with the size "provided_data_sectors" @@ -93,31 +102,27 @@ journal_sectors:number device. If the device is already formatted, the value from the superblock is used. -interleave_sectors:number +interleave_sectors:number (default 32768) The number of interleaved sectors. This values is rounded down to a power of two. If the device is already formatted, the value from the superblock is used. meta_device:device - Don't interleave the data and metadata on on device. Use a + Don't interleave the data and metadata on the device. Use a separate device for metadata. -buffer_sectors:number - The number of sectors in one buffer. The value is rounded down to - a power of two. - - The tag area is accessed using buffers, the buffer size is - configurable. The large buffer size means that the I/O size will - be larger, but there could be less I/Os issued. +buffer_sectors:number (default 128) + The number of sectors in one metadata buffer. The value is rounded + down to a power of two. -journal_watermark:number +journal_watermark:number (default 50) The journal watermark in percents. When the size of the journal exceeds this watermark, the thread that flushes the journal will be started. -commit_time:number +commit_time:number (default 10000) Commit time in milliseconds. When this time passes, the journal is - written. The journal is also written immediatelly if the FLUSH + written. The journal is also written immediately if the FLUSH request is received. internal_hash:algorithm(:key) (the key is optional) @@ -143,11 +148,11 @@ recalculate journal_crypt:algorithm(:key) (the key is optional) Encrypt the journal using given algorithm to make sure that the attacker can't read the journal. You can use a block cipher here - (such as "cbc(aes)") or a stream cipher (for example "chacha20", - "salsa20", "ctr(aes)" or "ecb(arc4)"). + (such as "cbc(aes)") or a stream cipher (for example "chacha20" + or "ctr(aes)"). The journal contains history of last writes to the block device, - an attacker reading the journal could see the last sector nubmers + an attacker reading the journal could see the last sector numbers that were written. From the sector numbers, the attacker can infer the size of files that were written. To protect against this situation, you can encrypt the journal. @@ -163,11 +168,10 @@ journal_mac:algorithm(:key) (the key is optional) the journal. Thus, modified sector number would be detected at this stage. -block_size:number - The size of a data block in bytes. The larger the block size the +block_size:number (default 512) + The size of a data block in bytes. The larger the block size the less overhead there is for per-block integrity metadata. - Supported values are 512, 1024, 2048 and 4096 bytes. If not - specified the default block size is 512 bytes. + Supported values are 512, 1024, 2048 and 4096 bytes. sectors_per_bit:number In the bitmap mode, this parameter specifies the number of @@ -177,17 +181,51 @@ bitmap_flush_interval:number The bitmap flush interval in milliseconds. The metadata buffers are synchronized when this interval expires. +allow_discards + Allow block discard requests (a.k.a. TRIM) for the integrity device. + Discards are only allowed to devices using internal hash. + fix_padding Use a smaller padding of the tag area that is more space-efficient. If this option is not present, large padding is used - that is for compatibility with older kernels. +fix_hmac + Improve security of internal_hash and journal_mac: + + - the section number is mixed to the mac, so that an attacker can't + copy sectors from one journal section to another journal section + - the superblock is protected by journal_mac + - a 16-byte salt stored in the superblock is mixed to the mac, so + that the attacker can't detect that two disks have the same hmac + key and also to disallow the attacker to move sectors from one + disk to another + +legacy_recalculate + Allow recalculating of volumes with HMAC keys. This is disabled by + default for security reasons - an attacker could modify the volume, + set recalc_sector to zero, and the kernel would not detect the + modification. + +The journal mode (D/J), buffer_sectors, journal_watermark, commit_time and +allow_discards can be changed when reloading the target (load an inactive +table and swap the tables with suspend and resume). The other arguments +should not be changed when reloading the target because the layout of disk +data depend on them and the reloaded target would be non-functional. + +For example, on a device using the default interleave_sectors of 32768, a +block_size of 512, and an internal_hash of crc32c with a tag size of 4 +bytes, it will take 128 KiB of tags to track a full data area, requiring +256 sectors of metadata per data area. With the default buffer_sectors of +128, that means there will be 2 buffers per metadata area, or 2 buffers +per 16 MiB of data. + +Status line: -The journal mode (D/J), buffer_sectors, journal_watermark, commit_time can -be changed when reloading the target (load an inactive table and swap the -tables with suspend and resume). The other arguments should not be changed -when reloading the target because the layout of disk data depend on them -and the reloaded target would be non-functional. +1. the number of integrity mismatches +2. provided data sectors - that is the number of sectors that the user + could use +3. the current recalculating position (or '-' if we didn't recalculate) The layout of the formatted block device: @@ -258,7 +296,8 @@ The layout of the formatted block device: Each run contains: * tag area - it contains integrity tags. There is one tag for each - sector in the data area + sector in the data area. The size of this area is always 4KiB or + greater. * data area - it contains data sectors. The number of data sectors in one run must be a power of two. log2 of this value is stored in the superblock. diff --git a/Documentation/admin-guide/device-mapper/dm-raid.rst b/Documentation/admin-guide/device-mapper/dm-raid.rst index f6344675e395..bb17e26e3c1b 100644 --- a/Documentation/admin-guide/device-mapper/dm-raid.rst +++ b/Documentation/admin-guide/device-mapper/dm-raid.rst @@ -71,7 +71,7 @@ The target is named "raid" and it accepts the following parameters:: ============= =============================================================== Reference: Chapter 4 of - http://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf + https://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf <#raid_params>: The number of parameters that follow. @@ -418,4 +418,6 @@ Version History specific devices are requested via rebuild. Fix RAID leg rebuild errors. 1.15.0 Fix size extensions not being synchronized in case of new MD bitmap - pages allocated; also fix those not occuring after previous reductions + pages allocated; also fix those not occurring after previous reductions + 1.15.1 Fix argument count and arguments for rebuild/write_mostly/journal_(dev|mode) + on the status line. diff --git a/Documentation/admin-guide/device-mapper/dm-zoned.rst b/Documentation/admin-guide/device-mapper/dm-zoned.rst index 07f56ebc1730..932383fe6e88 100644 --- a/Documentation/admin-guide/device-mapper/dm-zoned.rst +++ b/Documentation/admin-guide/device-mapper/dm-zoned.rst @@ -14,7 +14,7 @@ host-aware zoned block devices. For a more detailed description of the zoned block device models and their constraints see (for SCSI devices): -http://www.t10.org/drafts.htm#ZBC_Family +https://www.t10.org/drafts.htm#ZBC_Family and (for ATA devices): @@ -24,7 +24,7 @@ The dm-zoned implementation is simple and minimizes system overhead (CPU and memory usage as well as storage capacity loss). For a 10TB host-managed disk with 256 MB zones, dm-zoned memory usage per disk instance is at most 4.5 MB and as little as 5 zones will be used -internally for storing metadata and performaing reclaim operations. +internally for storing metadata and performing reclaim operations. dm-zoned target devices are formatted and checked using the dmzadm utility available at: @@ -37,12 +37,16 @@ Algorithm dm-zoned implements an on-disk buffering scheme to handle non-sequential write accesses to the sequential zones of a zoned block device. Conventional zones are used for caching as well as for storing internal -metadata. +metadata. It can also use a regular block device together with the zoned +block device; in that case the regular block device will be split logically +in zones with the same size as the zoned block device. These zones will be +placed in front of the zones from the zoned block device and will be handled +just like conventional zones. -The zones of the device are separated into 2 types: +The zones of the device(s) are separated into 2 types: 1) Metadata zones: these are conventional zones used to store metadata. -Metadata zones are not reported as useable capacity to the user. +Metadata zones are not reported as usable capacity to the user. 2) Data zones: all remaining zones, the vast majority of which will be sequential zones used exclusively to store user data. The conventional @@ -98,7 +102,7 @@ the buffer zone assigned. If the accessed chunk has no mapping, or the accessed blocks are invalid, the read buffer is zeroed and the read operation terminated. -After some time, the limited number of convnetional zones available may +After some time, the limited number of conventional zones available may be exhausted (all used to map chunks or buffer sequential zones) and unaligned writes to unbuffered chunks become impossible. To avoid this situation, a reclaim process regularly scans used conventional zones and @@ -127,6 +131,13 @@ resumed. Flushing metadata thus only temporarily delays write and discard requests. Read requests can be processed concurrently while metadata flush is being executed. +If a regular device is used in conjunction with the zoned block device, +a third set of metadata (without the zone bitmaps) is written to the +start of the zoned block device. This metadata has a generation counter of +'0' and will never be updated during normal operation; it just serves for +identification purposes. The first and second copy of the metadata +are located at the start of the regular block device. + Usage ===== @@ -138,9 +149,46 @@ Ex:: dmzadm --format /dev/sdxx -For a formatted device, the target can be created normally with the -dmsetup utility. The only parameter that dm-zoned requires is the -underlying zoned block device name. Ex:: - echo "0 `blockdev --getsize ${dev}` zoned ${dev}" | \ - dmsetup create dmz-`basename ${dev}` +If two drives are to be used, both devices must be specified, with the +regular block device as the first device. + +Ex:: + + dmzadm --format /dev/sdxx /dev/sdyy + + +Formatted device(s) can be started with the dmzadm utility, too.: + +Ex:: + + dmzadm --start /dev/sdxx /dev/sdyy + + +Information about the internal layout and current usage of the zones can +be obtained with the 'status' callback from dmsetup: + +Ex:: + + dmsetup status /dev/dm-X + +will return a line + + 0 <size> zoned <nr_zones> zones <nr_unmap_rnd>/<nr_rnd> random <nr_unmap_seq>/<nr_seq> sequential + +where <nr_zones> is the total number of zones, <nr_unmap_rnd> is the number +of unmapped (ie free) random zones, <nr_rnd> the total number of zones, +<nr_unmap_seq> the number of unmapped sequential zones, and <nr_seq> the +total number of sequential zones. + +Normally the reclaim process will be started once there are less than 50 +percent free random zones. In order to start the reclaim process manually +even before reaching this threshold the 'dmsetup message' function can be +used: + +Ex:: + + dmsetup message /dev/dm-X 0 reclaim + +will start the reclaim process and random zones will be moved to sequential +zones. diff --git a/Documentation/admin-guide/device-mapper/index.rst b/Documentation/admin-guide/device-mapper/index.rst index 4872fb6d2952..cc5aec861576 100644 --- a/Documentation/admin-guide/device-mapper/index.rst +++ b/Documentation/admin-guide/device-mapper/index.rst @@ -8,9 +8,12 @@ Device Mapper cache-policies cache delay + dm-clone dm-crypt dm-dust + dm-ebs dm-flakey + dm-ima dm-init dm-integrity dm-io @@ -31,6 +34,8 @@ Device Mapper switch thin-provisioning unstriped + vdo-design + vdo verity writecache zero diff --git a/Documentation/admin-guide/device-mapper/unstriped.rst b/Documentation/admin-guide/device-mapper/unstriped.rst index 0a8d3eb3f072..5772ccdd1f5f 100644 --- a/Documentation/admin-guide/device-mapper/unstriped.rst +++ b/Documentation/admin-guide/device-mapper/unstriped.rst @@ -35,7 +35,7 @@ An example of undoing an existing dm-stripe This small bash script will setup 4 loop devices and use the existing striped target to combine the 4 devices into one. It then will use -the unstriped target ontop of the striped device to access the +the unstriped target on top of the striped device to access the individual backing loop devices. We write data to the newly exposed unstriped devices and verify the data written matches the correct underlying device on the striped array:: @@ -110,8 +110,8 @@ to get a 92% reduction in read latency using this device mapper target. Example dmsetup usage ===================== -unstriped ontop of Intel NVMe device that has 2 cores ------------------------------------------------------ +unstriped on top of Intel NVMe device that has 2 cores +------------------------------------------------------ :: @@ -124,8 +124,8 @@ respectively:: /dev/mapper/nvmset0 /dev/mapper/nvmset1 -unstriped ontop of striped with 4 drives using 128K chunk size --------------------------------------------------------------- +unstriped on top of striped with 4 drives using 128K chunk size +--------------------------------------------------------------- :: diff --git a/Documentation/admin-guide/device-mapper/vdo-design.rst b/Documentation/admin-guide/device-mapper/vdo-design.rst new file mode 100644 index 000000000000..3cd59decbec0 --- /dev/null +++ b/Documentation/admin-guide/device-mapper/vdo-design.rst @@ -0,0 +1,633 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +================ +Design of dm-vdo +================ + +The dm-vdo (virtual data optimizer) target provides inline deduplication, +compression, zero-block elimination, and thin provisioning. A dm-vdo target +can be backed by up to 256TB of storage, and can present a logical size of +up to 4PB. This target was originally developed at Permabit Technology +Corp. starting in 2009. It was first released in 2013 and has been used in +production environments ever since. It was made open-source in 2017 after +Permabit was acquired by Red Hat. This document describes the design of +dm-vdo. For usage, see vdo.rst in the same directory as this file. + +Because deduplication rates fall drastically as the block size increases, a +vdo target has a maximum block size of 4K. However, it can achieve +deduplication rates of 254:1, i.e. up to 254 copies of a given 4K block can +reference a single 4K of actual storage. It can achieve compression rates +of 14:1. All zero blocks consume no storage at all. + +Theory of Operation +=================== + +The design of dm-vdo is based on the idea that deduplication is a two-part +problem. The first is to recognize duplicate data. The second is to avoid +storing multiple copies of those duplicates. Therefore, dm-vdo has two main +parts: a deduplication index (called UDS) that is used to discover +duplicate data, and a data store with a reference counted block map that +maps from logical block addresses to the actual storage location of the +data. + +Zones and Threading +------------------- + +Due to the complexity of data optimization, the number of metadata +structures involved in a single write operation to a vdo target is larger +than most other targets. Furthermore, because vdo must operate on small +block sizes in order to achieve good deduplication rates, acceptable +performance can only be achieved through parallelism. Therefore, vdo's +design attempts to be lock-free. + +Most of a vdo's main data structures are designed to be easily divided into +"zones" such that any given bio must only access a single zone of any zoned +structure. Safety with minimal locking is achieved by ensuring that during +normal operation, each zone is assigned to a specific thread, and only that +thread will access the portion of the data structure in that zone. +Associated with each thread is a work queue. Each bio is associated with a +request object (the "data_vio") which will be added to a work queue when +the next phase of its operation requires access to the structures in the +zone associated with that queue. + +Another way of thinking about this arrangement is that the work queue for +each zone has an implicit lock on the structures it manages for all its +operations, because vdo guarantees that no other thread will alter those +structures. + +Although each structure is divided into zones, this division is not +reflected in the on-disk representation of each data structure. Therefore, +the number of zones for each structure, and hence the number of threads, +can be reconfigured each time a vdo target is started. + +The Deduplication Index +----------------------- + +In order to identify duplicate data efficiently, vdo was designed to +leverage some common characteristics of duplicate data. From empirical +observations, we gathered two key insights. The first is that in most data +sets with significant amounts of duplicate data, the duplicates tend to +have temporal locality. When a duplicate appears, it is more likely that +other duplicates will be detected, and that those duplicates will have been +written at about the same time. This is why the index keeps records in +temporal order. The second insight is that new data is more likely to +duplicate recent data than it is to duplicate older data and in general, +there are diminishing returns to looking further back in time. Therefore, +when the index is full, it should cull its oldest records to make space for +new ones. Another important idea behind the design of the index is that the +ultimate goal of deduplication is to reduce storage costs. Since there is a +trade-off between the storage saved and the resources expended to achieve +those savings, vdo does not attempt to find every last duplicate block. It +is sufficient to find and eliminate most of the redundancy. + +Each block of data is hashed to produce a 16-byte block name. An index +record consists of this block name paired with the presumed location of +that data on the underlying storage. However, it is not possible to +guarantee that the index is accurate. In the most common case, this occurs +because it is too costly to update the index when a block is over-written +or discarded. Doing so would require either storing the block name along +with the blocks, which is difficult to do efficiently in block-based +storage, or reading and rehashing each block before overwriting it. +Inaccuracy can also result from a hash collision where two different blocks +have the same name. In practice, this is extremely unlikely, but because +vdo does not use a cryptographic hash, a malicious workload could be +constructed. Because of these inaccuracies, vdo treats the locations in the +index as hints, and reads each indicated block to verify that it is indeed +a duplicate before sharing the existing block with a new one. + +Records are collected into groups called chapters. New records are added to +the newest chapter, called the open chapter. This chapter is stored in a +format optimized for adding and modifying records, and the content of the +open chapter is not finalized until it runs out of space for new records. +When the open chapter fills up, it is closed and a new open chapter is +created to collect new records. + +Closing a chapter converts it to a different format which is optimized for +reading. The records are written to a series of record pages based on the +order in which they were received. This means that records with temporal +locality should be on a small number of pages, reducing the I/O required to +retrieve them. The chapter also compiles an index that indicates which +record page contains any given name. This index means that a request for a +name can determine exactly which record page may contain that record, +without having to load the entire chapter from storage. This index uses +only a subset of the block name as its key, so it cannot guarantee that an +index entry refers to the desired block name. It can only guarantee that if +there is a record for this name, it will be on the indicated page. Closed +chapters are read-only structures and their contents are never altered in +any way. + +Once enough records have been written to fill up all the available index +space, the oldest chapter is removed to make space for new chapters. Any +time a request finds a matching record in the index, that record is copied +into the open chapter. This ensures that useful block names remain available +in the index, while unreferenced block names are forgotten over time. + +In order to find records in older chapters, the index also maintains a +higher level structure called the volume index, which contains entries +mapping each block name to the chapter containing its newest record. This +mapping is updated as records for the block name are copied or updated, +ensuring that only the newest record for a given block name can be found. +An older record for a block name will no longer be found even though it has +not been deleted from its chapter. Like the chapter index, the volume index +uses only a subset of the block name as its key and can not definitively +say that a record exists for a name. It can only say which chapter would +contain the record if a record exists. The volume index is stored entirely +in memory and is saved to storage only when the vdo target is shut down. + +From the viewpoint of a request for a particular block name, it will first +look up the name in the volume index. This search will either indicate that +the name is new, or which chapter to search. If it returns a chapter, the +request looks up its name in the chapter index. This will indicate either +that the name is new, or which record page to search. Finally, if it is not +new, the request will look for its name in the indicated record page. +This process may require up to two page reads per request (one for the +chapter index page and one for the request page). However, recently +accessed pages are cached so that these page reads can be amortized across +many block name requests. + +The volume index and the chapter indexes are implemented using a +memory-efficient structure called a delta index. Instead of storing the +entire block name (the key) for each entry, the entries are sorted by name +and only the difference between adjacent keys (the delta) is stored. +Because we expect the hashes to be randomly distributed, the size of the +deltas follows an exponential distribution. Because of this distribution, +the deltas are expressed using a Huffman code to take up even less space. +The entire sorted list of keys is called a delta list. This structure +allows the index to use many fewer bytes per entry than a traditional hash +table, but it is slightly more expensive to look up entries, because a +request must read every entry in a delta list to add up the deltas in order +to find the record it needs. The delta index reduces this lookup cost by +splitting its key space into many sub-lists, each starting at a fixed key +value, so that each individual list is short. + +The default index size can hold 64 million records, corresponding to about +256GB of data. This means that the index can identify duplicate data if the +original data was written within the last 256GB of writes. This range is +called the deduplication window. If new writes duplicate data that is older +than that, the index will not be able to find it because the records of the +older data have been removed. This means that if an application writes a +200 GB file to a vdo target and then immediately writes it again, the two +copies will deduplicate perfectly. Doing the same with a 500 GB file will +result in no deduplication, because the beginning of the file will no +longer be in the index by the time the second write begins (assuming there +is no duplication within the file itself). + +If an application anticipates a data workload that will see useful +deduplication beyond the 256GB threshold, vdo can be configured to use a +larger index with a correspondingly larger deduplication window. (This +configuration can only be set when the target is created, not altered +later. It is important to consider the expected workload for a vdo target +before configuring it.) There are two ways to do this. + +One way is to increase the memory size of the index, which also increases +the amount of backing storage required. Doubling the size of the index will +double the length of the deduplication window at the expense of doubling +the storage size and the memory requirements. + +The other option is to enable sparse indexing. Sparse indexing increases +the deduplication window by a factor of 10, at the expense of also +increasing the storage size by a factor of 10. However with sparse +indexing, the memory requirements do not increase. The trade-off is +slightly more computation per request and a slight decrease in the amount +of deduplication detected. For most workloads with significant amounts of +duplicate data, sparse indexing will detect 97-99% of the deduplication +that a standard index will detect. + +The vio and data_vio Structures +------------------------------- + +A vio (short for Vdo I/O) is conceptually similar to a bio, with additional +fields and data to track vdo-specific information. A struct vio maintains a +pointer to a bio but also tracks other fields specific to the operation of +vdo. The vio is kept separate from its related bio because there are many +circumstances where vdo completes the bio but must continue to do work +related to deduplication or compression. + +Metadata reads and writes, and other writes that originate within vdo, use +a struct vio directly. Application reads and writes use a larger structure +called a data_vio to track information about their progress. A struct +data_vio contain a struct vio and also includes several other fields +related to deduplication and other vdo features. The data_vio is the +primary unit of application work in vdo. Each data_vio proceeds through a +set of steps to handle the application data, after which it is reset and +returned to a pool of data_vios for reuse. + +There is a fixed pool of 2048 data_vios. This number was chosen to bound +the amount of work that is required to recover from a crash. In addition, +benchmarks have indicated that increasing the size of the pool does not +significantly improve performance. + +The Data Store +-------------- + +The data store is implemented by three main data structures, all of which +work in concert to reduce or amortize metadata updates across as many data +writes as possible. + +*The Slab Depot* + +Most of the vdo volume belongs to the slab depot. The depot contains a +collection of slabs. The slabs can be up to 32GB, and are divided into +three sections. Most of a slab consists of a linear sequence of 4K blocks. +These blocks are used either to store data, or to hold portions of the +block map (see below). In addition to the data blocks, each slab has a set +of reference counters, using 1 byte for each data block. Finally each slab +has a journal. + +Reference updates are written to the slab journal. Slab journal blocks are +written out either when they are full, or when the recovery journal +requests they do so in order to allow the main recovery journal (see below) +to free up space. The slab journal is used both to ensure that the main +recovery journal can regularly free up space, and also to amortize the cost +of updating individual reference blocks. The reference counters are kept in +memory and are written out, a block at a time in oldest-dirtied-order, only +when there is a need to reclaim slab journal space. The write operations +are performed in the background as needed so they do not add latency to +particular I/O operations. + +Each slab is independent of every other. They are assigned to "physical +zones" in round-robin fashion. If there are P physical zones, then slab n +is assigned to zone n mod P. + +The slab depot maintains an additional small data structure, the "slab +summary," which is used to reduce the amount of work needed to come back +online after a crash. The slab summary maintains an entry for each slab +indicating whether or not the slab has ever been used, whether all of its +reference count updates have been persisted to storage, and approximately +how full it is. During recovery, each physical zone will attempt to recover +at least one slab, stopping whenever it has recovered a slab which has some +free blocks. Once each zone has some space, or has determined that none is +available, the target can resume normal operation in a degraded mode. Read +and write requests can be serviced, perhaps with degraded performance, +while the remainder of the dirty slabs are recovered. + +*The Block Map* + +The block map contains the logical to physical mapping. It can be thought +of as an array with one entry per logical address. Each entry is 5 bytes, +36 bits of which contain the physical block number which holds the data for +the given logical address. The other 4 bits are used to indicate the nature +of the mapping. Of the 16 possible states, one represents a logical address +which is unmapped (i.e. it has never been written, or has been discarded), +one represents an uncompressed block, and the other 14 states are used to +indicate that the mapped data is compressed, and which of the compression +slots in the compressed block contains the data for this logical address. + +In practice, the array of mapping entries is divided into "block map +pages," each of which fits in a single 4K block. Each block map page +consists of a header and 812 mapping entries. Each mapping page is actually +a leaf of a radix tree which consists of block map pages at each level. +There are 60 radix trees which are assigned to "logical zones" in round +robin fashion. (If there are L logical zones, tree n will belong to zone n +mod L.) At each level, the trees are interleaved, so logical addresses +0-811 belong to tree 0, logical addresses 812-1623 belong to tree 1, and so +on. The interleaving is maintained all the way up to the 60 root nodes. +Choosing 60 trees results in an evenly distributed number of trees per zone +for a large number of possible logical zone counts. The storage for the 60 +tree roots is allocated at format time. All other block map pages are +allocated out of the slabs as needed. This flexible allocation avoids the +need to pre-allocate space for the entire set of logical mappings and also +makes growing the logical size of a vdo relatively easy. + +In operation, the block map maintains two caches. It is prohibitive to keep +the entire leaf level of the trees in memory, so each logical zone +maintains its own cache of leaf pages. The size of this cache is +configurable at target start time. The second cache is allocated at start +time, and is large enough to hold all the non-leaf pages of the entire +block map. This cache is populated as pages are needed. + +*The Recovery Journal* + +The recovery journal is used to amortize updates across the block map and +slab depot. Each write request causes an entry to be made in the journal. +Entries are either "data remappings" or "block map remappings." For a data +remapping, the journal records the logical address affected and its old and +new physical mappings. For a block map remapping, the journal records the +block map page number and the physical block allocated for it. Block map +pages are never reclaimed or repurposed, so the old mapping is always 0. + +Each journal entry is an intent record summarizing the metadata updates +that are required for a data_vio. The recovery journal issues a flush +before each journal block write to ensure that the physical data for the +new block mappings in that block are stable on storage, and journal block +writes are all issued with the FUA bit set to ensure the recovery journal +entries themselves are stable. The journal entry and the data write it +represents must be stable on disk before the other metadata structures may +be updated to reflect the operation. These entries allow the vdo device to +reconstruct the logical to physical mappings after an unexpected +interruption such as a loss of power. + +*Write Path* + +All write I/O to vdo is asynchronous. Each bio will be acknowledged as soon +as vdo has done enough work to guarantee that it can complete the write +eventually. Generally, the data for acknowledged but unflushed write I/O +can be treated as though it is cached in memory. If an application +requires data to be stable on storage, it must issue a flush or write the +data with the FUA bit set like any other asynchronous I/O. Shutting down +the vdo target will also flush any remaining I/O. + +Application write bios follow the steps outlined below. + +1. A data_vio is obtained from the data_vio pool and associated with the + application bio. If there are no data_vios available, the incoming bio + will block until a data_vio is available. This provides back pressure + to the application. The data_vio pool is protected by a spin lock. + + The newly acquired data_vio is reset and the bio's data is copied into + the data_vio if it is a write and the data is not all zeroes. The data + must be copied because the application bio can be acknowledged before + the data_vio processing is complete, which means later processing steps + will no longer have access to the application bio. The application bio + may also be smaller than 4K, in which case the data_vio will have + already read the underlying block and the data is instead copied over + the relevant portion of the larger block. + +2. The data_vio places a claim (the "logical lock") on the logical address + of the bio. It is vital to prevent simultaneous modifications of the + same logical address, because deduplication involves sharing blocks. + This claim is implemented as an entry in a hashtable where the key is + the logical address and the value is a pointer to the data_vio + currently handling that address. + + If a data_vio looks in the hashtable and finds that another data_vio is + already operating on that logical address, it waits until the previous + operation finishes. It also sends a message to inform the current + lock holder that it is waiting. Most notably, a new data_vio waiting + for a logical lock will flush the previous lock holder out of the + compression packer (step 8d) rather than allowing it to continue + waiting to be packed. + + This stage requires the data_vio to get an implicit lock on the + appropriate logical zone to prevent concurrent modifications of the + hashtable. This implicit locking is handled by the zone divisions + described above. + +3. The data_vio traverses the block map tree to ensure that all the + necessary internal tree nodes have been allocated, by trying to find + the leaf page for its logical address. If any interior tree page is + missing, it is allocated at this time out of the same physical storage + pool used to store application data. + + a. If any page-node in the tree has not yet been allocated, it must be + allocated before the write can continue. This step requires the + data_vio to lock the page-node that needs to be allocated. This + lock, like the logical block lock in step 2, is a hashtable entry + that causes other data_vios to wait for the allocation process to + complete. + + The implicit logical zone lock is released while the allocation is + happening, in order to allow other operations in the same logical + zone to proceed. The details of allocation are the same as in + step 4. Once a new node has been allocated, that node is added to + the tree using a similar process to adding a new data block mapping. + The data_vio journals the intent to add the new node to the block + map tree (step 10), updates the reference count of the new block + (step 11), and reacquires the implicit logical zone lock to add the + new mapping to the parent tree node (step 12). Once the tree is + updated, the data_vio proceeds down the tree. Any other data_vios + waiting on this allocation also proceed. + + b. In the steady-state case, the block map tree nodes will already be + allocated, so the data_vio just traverses the tree until it finds + the required leaf node. The location of the mapping (the "block map + slot") is recorded in the data_vio so that later steps do not need + to traverse the tree again. The data_vio then releases the implicit + logical zone lock. + +4. If the block is a zero block, skip to step 9. Otherwise, an attempt is + made to allocate a free data block. This allocation ensures that the + data_vio can write its data somewhere even if deduplication and + compression are not possible. This stage gets an implicit lock on a + physical zone to search for free space within that zone. + + The data_vio will search each slab in a zone until it finds a free + block or decides there are none. If the first zone has no free space, + it will proceed to search the next physical zone by taking the implicit + lock for that zone and releasing the previous one until it finds a + free block or runs out of zones to search. The data_vio will acquire a + struct pbn_lock (the "physical block lock") on the free block. The + struct pbn_lock also has several fields to record the various kinds of + claims that data_vios can have on physical blocks. The pbn_lock is + added to a hashtable like the logical block locks in step 2. This + hashtable is also covered by the implicit physical zone lock. The + reference count of the free block is updated to prevent any other + data_vio from considering it free. The reference counters are a + sub-component of the slab and are thus also covered by the implicit + physical zone lock. + +5. If an allocation was obtained, the data_vio has all the resources it + needs to complete the write. The application bio can safely be + acknowledged at this point. The acknowledgment happens on a separate + thread to prevent the application callback from blocking other data_vio + operations. + + If an allocation could not be obtained, the data_vio continues to + attempt to deduplicate or compress the data, but the bio is not + acknowledged because the vdo device may be out of space. + +6. At this point vdo must determine where to store the application data. + The data_vio's data is hashed and the hash (the "record name") is + recorded in the data_vio. + +7. The data_vio reserves or joins a struct hash_lock, which manages all of + the data_vios currently writing the same data. Active hash locks are + tracked in a hashtable similar to the way logical block locks are + tracked in step 2. This hashtable is covered by the implicit lock on + the hash zone. + + If there is no existing hash lock for this data_vio's record_name, the + data_vio obtains a hash lock from the pool, adds it to the hashtable, + and sets itself as the new hash lock's "agent." The hash_lock pool is + also covered by the implicit hash zone lock. The hash lock agent will + do all the work to decide where the application data will be + written. If a hash lock for the data_vio's record_name already exists, + and the data_vio's data is the same as the agent's data, the new + data_vio will wait for the agent to complete its work and then share + its result. + + In the rare case that a hash lock exists for the data_vio's hash but + the data does not match the hash lock's agent, the data_vio skips to + step 8h and attempts to write its data directly. This can happen if two + different data blocks produce the same hash, for example. + +8. The hash lock agent attempts to deduplicate or compress its data with + the following steps. + + a. The agent initializes and sends its embedded deduplication request + (struct uds_request) to the deduplication index. This does not + require the data_vio to get any locks because the index components + manage their own locking. The data_vio waits until it either gets a + response from the index or times out. + + b. If the deduplication index returns advice, the data_vio attempts to + obtain a physical block lock on the indicated physical address, in + order to read the data and verify that it is the same as the + data_vio's data, and that it can accept more references. If the + physical address is already locked by another data_vio, the data at + that address may soon be overwritten so it is not safe to use the + address for deduplication. + + c. If the data matches and the physical block can add references, the + agent and any other data_vios waiting on it will record this + physical block as their new physical address and proceed to step 9 + to record their new mapping. If there are more data_vios in the hash + lock than there are references available, one of the remaining + data_vios becomes the new agent and continues to step 8d as if no + valid advice was returned. + + d. If no usable duplicate block was found, the agent first checks that + it has an allocated physical block (from step 3) that it can write + to. If the agent does not have an allocation, some other data_vio in + the hash lock that does have an allocation takes over as agent. If + none of the data_vios have an allocated physical block, these writes + are out of space, so they proceed to step 13 for cleanup. + + e. The agent attempts to compress its data. If the data does not + compress, the data_vio will continue to step 8h to write its data + directly. + + If the compressed size is small enough, the agent will release the + implicit hash zone lock and go to the packer (struct packer) where + it will be placed in a bin (struct packer_bin) along with other + data_vios. All compression operations require the implicit lock on + the packer zone. + + The packer can combine up to 14 compressed blocks in a single 4k + data block. Compression is only helpful if vdo can pack at least 2 + data_vios into a single data block. This means that a data_vio may + wait in the packer for an arbitrarily long time for other data_vios + to fill out the compressed block. There is a mechanism for vdo to + evict waiting data_vios when continuing to wait would cause + problems. Circumstances causing an eviction include an application + flush, device shutdown, or a subsequent data_vio trying to overwrite + the same logical block address. A data_vio may also be evicted from + the packer if it cannot be paired with any other compressed block + before more compressible blocks need to use its bin. An evicted + data_vio will proceed to step 8h to write its data directly. + + f. If the agent fills a packer bin, either because all 14 of its slots + are used or because it has no remaining space, it is written out + using the allocated physical block from one of its data_vios. Step + 8d has already ensured that an allocation is available. + + g. Each data_vio sets the compressed block as its new physical address. + The data_vio obtains an implicit lock on the physical zone and + acquires the struct pbn_lock for the compressed block, which is + modified to be a shared lock. Then it releases the implicit physical + zone lock and proceeds to step 8i. + + h. Any data_vio evicted from the packer will have an allocation from + step 3. It will write its data to that allocated physical block. + + i. After the data is written, if the data_vio is the agent of a hash + lock, it will reacquire the implicit hash zone lock and share its + physical address with as many other data_vios in the hash lock as + possible. Each data_vio will then proceed to step 9 to record its + new mapping. + + j. If the agent actually wrote new data (whether compressed or not), + the deduplication index is updated to reflect the location of the + new data. The agent then releases the implicit hash zone lock. + +9. The data_vio determines the previous mapping of the logical address. + There is a cache for block map leaf pages (the "block map cache"), + because there are usually too many block map leaf nodes to store + entirely in memory. If the desired leaf page is not in the cache, the + data_vio will reserve a slot in the cache and load the desired page + into it, possibly evicting an older cached page. The data_vio then + finds the current physical address for this logical address (the "old + physical mapping"), if any, and records it. This step requires a lock + on the block map cache structures, covered by the implicit logical zone + lock. + +10. The data_vio makes an entry in the recovery journal containing the + logical block address, the old physical mapping, and the new physical + mapping. Making this journal entry requires holding the implicit + recovery journal lock. The data_vio will wait in the journal until all + recovery blocks up to the one containing its entry have been written + and flushed to ensure the transaction is stable on storage. + +11. Once the recovery journal entry is stable, the data_vio makes two slab + journal entries: an increment entry for the new mapping, and a + decrement entry for the old mapping. These two operations each require + holding a lock on the affected physical slab, covered by its implicit + physical zone lock. For correctness during recovery, the slab journal + entries in any given slab journal must be in the same order as the + corresponding recovery journal entries. Therefore, if the two entries + are in different zones, they are made concurrently, and if they are in + the same zone, the increment is always made before the decrement in + order to avoid underflow. After each slab journal entry is made in + memory, the associated reference count is also updated in memory. + +12. Once both of the reference count updates are done, the data_vio + acquires the implicit logical zone lock and updates the + logical-to-physical mapping in the block map to point to the new + physical block. At this point the write operation is complete. + +13. If the data_vio has a hash lock, it acquires the implicit hash zone + lock and releases its hash lock to the pool. + + The data_vio then acquires the implicit physical zone lock and releases + the struct pbn_lock it holds for its allocated block. If it had an + allocation that it did not use, it also sets the reference count for + that block back to zero to free it for use by subsequent data_vios. + + The data_vio then acquires the implicit logical zone lock and releases + the logical block lock acquired in step 2. + + The application bio is then acknowledged if it has not previously been + acknowledged, and the data_vio is returned to the pool. + +*Read Path* + +An application read bio follows a much simpler set of steps. It does steps +1 and 2 in the write path to obtain a data_vio and lock its logical +address. If there is already a write data_vio in progress for that logical +address that is guaranteed to complete, the read data_vio will copy the +data from the write data_vio and return it. Otherwise, it will look up the +logical-to-physical mapping by traversing the block map tree as in step 3, +and then read and possibly decompress the indicated data at the indicated +physical block address. A read data_vio will not allocate block map tree +nodes if they are missing. If the interior block map nodes do not exist +yet, the logical block map address must still be unmapped and the read +data_vio will return all zeroes. A read data_vio handles cleanup and +acknowledgment as in step 13, although it only needs to release the logical +lock and return itself to the pool. + +*Small Writes* + +All storage within vdo is managed as 4KB blocks, but it can accept writes +as small as 512 bytes. Processing a write that is smaller than 4K requires +a read-modify-write operation that reads the relevant 4K block, copies the +new data over the approriate sectors of the block, and then launches a +write operation for the modified data block. The read and write stages of +this operation are nearly identical to the normal read and write +operations, and a single data_vio is used throughout this operation. + +*Recovery* + +When a vdo is restarted after a crash, it will attempt to recover from the +recovery journal. During the pre-resume phase of the next start, the +recovery journal is read. The increment portion of valid entries are played +into the block map. Next, valid entries are played, in order as required, +into the slab journals. Finally, each physical zone attempts to replay at +least one slab journal to reconstruct the reference counts of one slab. +Once each zone has some free space (or has determined that it has none), +the vdo comes back online, while the remainder of the slab journals are +used to reconstruct the rest of the reference counts in the background. + +*Read-only Rebuild* + +If a vdo encounters an unrecoverable error, it will enter read-only mode. +This mode indicates that some previously acknowledged data may have been +lost. The vdo may be instructed to rebuild as best it can in order to +return to a writable state. However, this is never done automatically due +to the possibility that data has been lost. During a read-only rebuild, the +block map is recovered from the recovery journal as before. However, the +reference counts are not rebuilt from the slab journals. Instead, the +reference counts are zeroed, the entire block map is traversed, and the +reference counts are updated from the block mappings. While this may lose +some data, it ensures that the block map and reference counts are +consistent with each other. This allows vdo to resume normal operation and +accept further writes. diff --git a/Documentation/admin-guide/device-mapper/vdo.rst b/Documentation/admin-guide/device-mapper/vdo.rst new file mode 100644 index 000000000000..7e1ecafdf91e --- /dev/null +++ b/Documentation/admin-guide/device-mapper/vdo.rst @@ -0,0 +1,406 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +dm-vdo +====== + +The dm-vdo (virtual data optimizer) device mapper target provides +block-level deduplication, compression, and thin provisioning. As a device +mapper target, it can add these features to the storage stack, compatible +with any file system. The vdo target does not protect against data +corruption, relying instead on integrity protection of the storage below +it. It is strongly recommended that lvm be used to manage vdo volumes. See +lvmvdo(7). + +Userspace component +=================== + +Formatting a vdo volume requires the use of the 'vdoformat' tool, available +at: + +https://github.com/dm-vdo/vdo/ + +In most cases, a vdo target will recover from a crash automatically the +next time it is started. In cases where it encountered an unrecoverable +error (either during normal operation or crash recovery) the target will +enter or come up in read-only mode. Because read-only mode is indicative of +data-loss, a positive action must be taken to bring vdo out of read-only +mode. The 'vdoforcerebuild' tool, available from the same repo, is used to +prepare a read-only vdo to exit read-only mode. After running this tool, +the vdo target will rebuild its metadata the next time it is +started. Although some data may be lost, the rebuilt vdo's metadata will be +internally consistent and the target will be writable again. + +The repo also contains additional userspace tools which can be used to +inspect a vdo target's on-disk metadata. Fortunately, these tools are +rarely needed except by dm-vdo developers. + +Metadata requirements +===================== + +Each vdo volume reserves 3GB of space for metadata, or more depending on +its configuration. It is helpful to check that the space saved by +deduplication and compression is not cancelled out by the metadata +requirements. An estimation of the space saved for a specific dataset can +be computed with the vdo estimator tool, which is available at: + +https://github.com/dm-vdo/vdoestimator/ + +Target interface +================ + +Table line +---------- + +:: + + <offset> <logical device size> vdo V4 <storage device> + <storage device size> <minimum I/O size> <block map cache size> + <block map era length> [optional arguments] + + +Required parameters: + + offset: + The offset, in sectors, at which the vdo volume's logical + space begins. + + logical device size: + The size of the device which the vdo volume will service, + in sectors. Must match the current logical size of the vdo + volume. + + storage device: + The device holding the vdo volume's data and metadata. + + storage device size: + The size of the device holding the vdo volume, as a number + of 4096-byte blocks. Must match the current size of the vdo + volume. + + minimum I/O size: + The minimum I/O size for this vdo volume to accept, in + bytes. Valid values are 512 or 4096. The recommended value + is 4096. + + block map cache size: + The size of the block map cache, as a number of 4096-byte + blocks. The minimum and recommended value is 32768 blocks. + If the logical thread count is non-zero, the cache size + must be at least 4096 blocks per logical thread. + + block map era length: + The speed with which the block map cache writes out + modified block map pages. A smaller era length is likely to + reduce the amount of time spent rebuilding, at the cost of + increased block map writes during normal operation. The + maximum and recommended value is 16380; the minimum value + is 1. + +Optional parameters: +-------------------- +Some or all of these parameters may be specified as <key> <value> pairs. + +Thread related parameters: + +Different categories of work are assigned to separate thread groups, and +the number of threads in each group can be configured separately. + +If <hash>, <logical>, and <physical> are all set to 0, the work handled by +all three thread types will be handled by a single thread. If any of these +values are non-zero, all of them must be non-zero. + + ack: + The number of threads used to complete bios. Since + completing a bio calls an arbitrary completion function + outside the vdo volume, threads of this type allow the vdo + volume to continue processing requests even when bio + completion is slow. The default is 1. + + bio: + The number of threads used to issue bios to the underlying + storage. Threads of this type allow the vdo volume to + continue processing requests even when bio submission is + slow. The default is 4. + + bioRotationInterval: + The number of bios to enqueue on each bio thread before + switching to the next thread. The value must be greater + than 0 and not more than 1024; the default is 64. + + cpu: + The number of threads used to do CPU-intensive work, such + as hashing and compression. The default is 1. + + hash: + The number of threads used to manage data comparisons for + deduplication based on the hash value of data blocks. The + default is 0. + + logical: + The number of threads used to manage caching and locking + based on the logical address of incoming bios. The default + is 0; the maximum is 60. + + physical: + The number of threads used to manage administration of the + underlying storage device. At format time, a slab size for + the vdo is chosen; the vdo storage device must be large + enough to have at least 1 slab per physical thread. The + default is 0; the maximum is 16. + +Miscellaneous parameters: + + maxDiscard: + The maximum size of discard bio accepted, in 4096-byte + blocks. I/O requests to a vdo volume are normally split + into 4096-byte blocks, and processed up to 2048 at a time. + However, discard requests to a vdo volume can be + automatically split to a larger size, up to <maxDiscard> + 4096-byte blocks in a single bio, and are limited to 1500 + at a time. Increasing this value may provide better overall + performance, at the cost of increased latency for the + individual discard requests. The default and minimum is 1; + the maximum is UINT_MAX / 4096. + + deduplication: + Whether deduplication is enabled. The default is 'on'; the + acceptable values are 'on' and 'off'. + + compression: + Whether compression is enabled. The default is 'off'; the + acceptable values are 'on' and 'off'. + +Device modification +------------------- + +A modified table may be loaded into a running, non-suspended vdo volume. +The modifications will take effect when the device is next resumed. The +modifiable parameters are <logical device size>, <physical device size>, +<maxDiscard>, <compression>, and <deduplication>. + +If the logical device size or physical device size are changed, upon +successful resume vdo will store the new values and require them on future +startups. These two parameters may not be decreased. The logical device +size may not exceed 4 PB. The physical device size must increase by at +least 32832 4096-byte blocks if at all, and must not exceed the size of the +underlying storage device. Additionally, when formatting the vdo device, a +slab size is chosen: the physical device size may never increase above the +size which provides 8192 slabs, and each increase must be large enough to +add at least one new slab. + +Examples: + +Start a previously-formatted vdo volume with 1 GB logical space and 1 GB +physical space, storing to /dev/dm-1 which has more than 1 GB of space. + +:: + + dmsetup create vdo0 --table \ + "0 2097152 vdo V4 /dev/dm-1 262144 4096 32768 16380" + +Grow the logical size to 4 GB. + +:: + + dmsetup reload vdo0 --table \ + "0 8388608 vdo V4 /dev/dm-1 262144 4096 32768 16380" + dmsetup resume vdo0 + +Grow the physical size to 2 GB. + +:: + + dmsetup reload vdo0 --table \ + "0 8388608 vdo V4 /dev/dm-1 524288 4096 32768 16380" + dmsetup resume vdo0 + +Grow the physical size by 1 GB more and increase max discard sectors. + +:: + + dmsetup reload vdo0 --table \ + "0 10485760 vdo V4 /dev/dm-1 786432 4096 32768 16380 maxDiscard 8" + dmsetup resume vdo0 + +Stop the vdo volume. + +:: + + dmsetup remove vdo0 + +Start the vdo volume again. Note that the logical and physical device sizes +must still match, but other parameters can change. + +:: + + dmsetup create vdo1 --table \ + "0 10485760 vdo V4 /dev/dm-1 786432 512 65550 5000 hash 1 logical 3 physical 2" + +Messages +-------- +All vdo devices accept messages in the form: + +:: + dmsetup message <target-name> 0 <message-name> <message-parameters> + +The messages are: + + stats: + Outputs the current view of the vdo statistics. Mostly used + by the vdostats userspace program to interpret the output + buffer. + + dump: + Dumps many internal structures to the system log. This is + not always safe to run, so it should only be used to debug + a hung vdo. Optional parameters to specify structures to + dump are: + + viopool: The pool of I/O requests incoming bios + pools: A synonym of 'viopool' + vdo: Most of the structures managing on-disk data + queues: Basic information about each vdo thread + threads: A synonym of 'queues' + default: Equivalent to 'queues vdo' + all: All of the above. + + dump-on-shutdown: + Perform a default dump next time vdo shuts down. + + +Status +------ + +:: + + <device> <operating mode> <in recovery> <index state> + <compression state> <physical blocks used> <total physical blocks> + + device: + The name of the vdo volume. + + operating mode: + The current operating mode of the vdo volume; values may be + 'normal', 'recovering' (the volume has detected an issue + with its metadata and is attempting to repair itself), and + 'read-only' (an error has occurred that forces the vdo + volume to only support read operations and not writes). + + in recovery: + Whether the vdo volume is currently in recovery mode; + values may be 'recovering' or '-' which indicates not + recovering. + + index state: + The current state of the deduplication index in the vdo + volume; values may be 'closed', 'closing', 'error', + 'offline', 'online', 'opening', and 'unknown'. + + compression state: + The current state of compression in the vdo volume; values + may be 'offline' and 'online'. + + used physical blocks: + The number of physical blocks in use by the vdo volume. + + total physical blocks: + The total number of physical blocks the vdo volume may use; + the difference between this value and the + <used physical blocks> is the number of blocks the vdo + volume has left before being full. + +Memory Requirements +=================== + +A vdo target requires a fixed 38 MB of RAM along with the following amounts +that scale with the target: + +- 1.15 MB of RAM for each 1 MB of configured block map cache size. The + block map cache requires a minimum of 150 MB. +- 1.6 MB of RAM for each 1 TB of logical space. +- 268 MB of RAM for each 1 TB of physical storage managed by the volume. + +The deduplication index requires additional memory which scales with the +size of the deduplication window. For dense indexes, the index requires 1 +GB of RAM per 1 TB of window. For sparse indexes, the index requires 1 GB +of RAM per 10 TB of window. The index configuration is set when the target +is formatted and may not be modified. + +Module Parameters +================= + +The vdo driver has a numeric parameter 'log_level' which controls the +verbosity of logging from the driver. The default setting is 6 +(LOGLEVEL_INFO and more severe messages). + +Run-time Usage +============== + +When using dm-vdo, it is important to be aware of the ways in which its +behavior differs from other storage targets. + +- There is no guarantee that over-writes of existing blocks will succeed. + Because the underlying storage may be multiply referenced, over-writing + an existing block generally requires a vdo to have a free block + available. + +- When blocks are no longer in use, sending a discard request for those + blocks lets the vdo release references for those blocks. If the vdo is + thinly provisioned, discarding unused blocks is essential to prevent the + target from running out of space. However, due to the sharing of + duplicate blocks, no discard request for any given logical block is + guaranteed to reclaim space. + +- Assuming the underlying storage properly implements flush requests, vdo + is resilient against crashes, however, unflushed writes may or may not + persist after a crash. + +- Each write to a vdo target entails a significant amount of processing. + However, much of the work is paralellizable. Therefore, vdo targets + achieve better throughput at higher I/O depths, and can support up 2048 + requests in parallel. + +Tuning +====== + +The vdo device has many options, and it can be difficult to make optimal +choices without perfect knowledge of the workload. Additionally, most +configuration options must be set when a vdo target is started, and cannot +be changed without shutting it down completely; the configuration cannot be +changed while the target is active. Ideally, tuning with simulated +workloads should be performed before deploying vdo in production +environments. + +The most important value to adjust is the block map cache size. In order to +service a request for any logical address, a vdo must load the portion of +the block map which holds the relevant mapping. These mappings are cached. +Performance will suffer when the working set does not fit in the cache. By +default, a vdo allocates 128 MB of metadata cache in RAM to support +efficient access to 100 GB of logical space at a time. It should be scaled +up proportionally for larger working sets. + +The logical and physical thread counts should also be adjusted. A logical +thread controls a disjoint section of the block map, so additional logical +threads increase parallelism and can increase throughput. Physical threads +control a disjoint section of the data blocks, so additional physical +threads can also increase throughput. However, excess threads can waste +resources and increase contention. + +Bio submission threads control the parallelism involved in sending I/O to +the underlying storage; fewer threads mean there is more opportunity to +reorder I/O requests for performance benefit, but also that each I/O +request has to wait longer before being submitted. + +Bio acknowledgment threads are used for finishing I/O requests. This is +done on dedicated threads since the amount of work required to execute a +bio's callback can not be controlled by the vdo itself. Usually one thread +is sufficient but additional threads may be beneficial, particularly when +bios have CPU-heavy callbacks. + +CPU threads are used for hashing and for compression; in workloads with +compression enabled, more threads may result in higher throughput. + +Hash threads are used to sort active requests by hash and determine whether +they should deduplicate; the most CPU intensive actions done by these +threads are comparison of 4096-byte data blocks. In most cases, a single +hash thread is sufficient. diff --git a/Documentation/admin-guide/device-mapper/verity.rst b/Documentation/admin-guide/device-mapper/verity.rst index bb02caa45289..a65c1602cb23 100644 --- a/Documentation/admin-guide/device-mapper/verity.rst +++ b/Documentation/admin-guide/device-mapper/verity.rst @@ -69,7 +69,7 @@ Construction Parameters <#opt_params> Number of optional parameters. If there are no optional parameters, - the optional paramaters section can be skipped or #opt_params can be zero. + the optional parameters section can be skipped or #opt_params can be zero. Otherwise #opt_params is the number of following arguments. Example of optional parameters section: @@ -83,6 +83,10 @@ restart_on_corruption not compatible with ignore_corruption and requires user space support to avoid restart loops. +panic_on_corruption + Panic the device when a corrupted block is discovered. This option is + not compatible with ignore_corruption and restart_on_corruption. + ignore_zero_blocks Do not verify blocks that are expected to contain zeroes and always return zeroes instead. This may be useful if the partition contains unused blocks @@ -130,7 +134,16 @@ root_hash_sig_key_desc <key_description> the pkcs7 signature of the roothash. The pkcs7 signature is used to validate the root hash during the creation of the device mapper block device. Verification of roothash depends on the config DM_VERITY_VERIFY_ROOTHASH_SIG - being set in the kernel. + being set in the kernel. The signatures are checked against the builtin + trusted keyring by default, or the secondary trusted keyring if + DM_VERITY_VERIFY_ROOTHASH_SIG_SECONDARY_KEYRING is set. The secondary + trusted keyring includes by default the builtin trusted keyring, and it can + also gain new certificates at run time if they are signed by a certificate + already in the secondary trusted keyring. + +try_verify_in_tasklet + If verity hashes are in cache, verify data blocks in kernel tasklet instead + of workqueue. This option can reduce IO latency. Theory of operation =================== diff --git a/Documentation/admin-guide/device-mapper/writecache.rst b/Documentation/admin-guide/device-mapper/writecache.rst index d3d7690f5e8d..60c16b7fd5ac 100644 --- a/Documentation/admin-guide/device-mapper/writecache.rst +++ b/Documentation/admin-guide/device-mapper/writecache.rst @@ -12,7 +12,6 @@ first sector should contain valid superblock from previous invocation. Constructor parameters: 1. type of the cache device - "p" or "s" - - p - persistent memory - s - SSD 2. the underlying device that will be cached @@ -37,10 +36,10 @@ Constructor parameters: autocommit_blocks n (default: 64 for pmem, 65536 for ssd) when the application writes this amount of blocks without issuing the FLUSH request, the blocks are automatically - commited + committed autocommit_time ms (default: 1000) autocommit time in milliseconds. The data is automatically - commited if this time passes and no FLUSH request is + committed if this time passes and no FLUSH request is received fua (by default on) applicable only to persistent memory - use the FUA flag @@ -53,19 +52,51 @@ Constructor parameters: - some underlying devices perform better with fua, some with nofua. The user should test it + cleaner + when this option is activated (either in the constructor + arguments or by a message), the cache will not promote + new writes (however, writes to already cached blocks are + promoted, to avoid data corruption due to misordered + writes) and it will gradually writeback any cached + data. The userspace can then monitor the cleaning + process with "dmsetup status". When the number of cached + blocks drops to zero, userspace can unload the + dm-writecache target and replace it with dm-linear or + other targets. + max_age n + specifies the maximum age of a block in milliseconds. If + a block is stored in the cache for too long, it will be + written to the underlying device and cleaned up. + metadata_only + only metadata is promoted to the cache. This option + improves performance for heavier REQ_META workloads. + pause_writeback n (default: 3000) + pause writeback if there was some write I/O redirected to + the origin volume in the last n milliseconds Status: + 1. error indicator - 0 if there was no error, otherwise error number 2. the number of blocks 3. the number of free blocks 4. the number of blocks under writeback +5. the number of read blocks +6. the number of read blocks that hit the cache +7. the number of write blocks +8. the number of write blocks that hit uncommitted block +9. the number of write blocks that hit committed block +10. the number of write blocks that bypass the cache +11. the number of write blocks that are allocated in the cache +12. the number of write requests that are blocked on the freelist +13. the number of flush requests +14. the number of discarded blocks Messages: flush - flush the cache device. The message returns successfully + Flush the cache device. The message returns successfully if the cache device was flushed without an error flush_on_suspend - flush the cache device on next suspend. Use this message + Flush the cache device on next suspend. Use this message when you are going to remove the cache device. The proper sequence for removing the cache device is: @@ -77,3 +108,7 @@ Messages: 5. resume the device, so that it will use the linear target 6. the cache device is now inactive and it can be deleted + cleaner + See above "cleaner" constructor documentation. + clear_stats + Clear the statistics that are reported on the status line |