Copy data between files or read data from a file. Specialized for "files"
that are storage devices, especially those that can use the SCSI command
sets (e.g. SATA and SAS disks, plus DVD drives). Can issue SCSI commands
in pass-through ("pt") mode. Similar syntax and semantics to the Unix
dd(1)
command.
For comparison, the SYNOPSIS section above shows both the
ddpt
command line options followed by GNU's
dd(1)
command line options. Broadly speaking ddpt can be considered a super-set
of dd. See the section on DD DIFFERENCES for significant differences
between ddpt and dd.
ddpt does a segmented copy, first reading in BPT*IBS bytes from
IFILE (or less if near the end of the copy) into a copy buffer. In the
absence of the various options and conditions that bypass the write
operation, the copy buffer is then written out to OFILE. The copy
process continues working its way along IFILE and OFILE until
either COUNT is exhausted, an end of file is detected, or an error
occurs. If IBS and OBS are different, ddpt restricts the value
of OBS such that the copy buffer is an integral number output
blocks (i.e. (((IBS*BPT) % OBS) == 0) ). In the following
descriptions, "segment" refers to all or part of a copy buffer.
The term "pt device" is used for a pass-through device to which SCSI
commands like READ(10) and WRITE(10) may be sent. A pt device may only be
able to process SCSI commands in which case the "pt" flag is assumed. The
ability to recognize such a pt only device may vary depending on the
operating system (e.g. in Linux '/dev/sg2' is recognized). However
if a device can process either normal UNIX read()/write() calls or
pass-through SCSI commands then the default is to use UNIX read()/write()
calls. That default can be overridden by using the "pt"
flag (e.g. "if=/dev/sdc iflag=pt"). When pt access is specified any
partition information is
ignored.
So "if=/dev/sdc2 iflag=pt skip=3" will start at logical block address 3
of '/dev/sdc'. As a protection measure in version 0.92 ddpt will only
accept that if the force flag is given in addition to pt.
OPTIONS
bpt=BPT[,OBPC]
where BPT is Blocks Per Transfer. The copy is made up of multiple
transfers, each first reading BPT input blocks (i.e. BPT*IBS
bytes) from IFILE into the copy buffer and then from that copy buffer
writing BPT*IBS/OBS output blocks to OFILE. This continues
until the copy is finished, with the last transfer being potentially
shorter. The default BPT value varies depending on IBS. When
IBS < 8, BPT is 8192; when IBS < 64, BPT is 1024;
when IBS < 1024, BPT is 128; when IBS < 8192, BPT
is 16; when IBS < 32768, BPT is 4; else BPT defaults
to 1. If BPT is given as 0 it is treated as the default value.
For "bs=512", BPT defaults to 128 so that 64 KiB (or less) is read
from IFILE into the copy buffer.
The optional OBPC (Output Blocks Per Check) argument controls
controls the granularity of sparse writes, write sparing and trim checks.
The default granularity is the size of the copy buffer (i.e. BPT*IBS
bytes). That can be reduced by specifying OBPC. The finest
granularity is when OBPC is 1 which implies the unit of each check
is OBS bytes. When OBPC is 0, or not given, the default
granularity is used. Large OBPC values are rounded down so that
OBPC*OBS does not exceed the size of the copy buffer.
bs=BS
where BS is the IFILE and OFILE block size in bytes.
Conflicts with either "ibs=" or "obs=" options. The value of BS
is placed in IBS and OBS.
If IFILE or OFILE is a "pt" device then BSmust
be the logical block size of the device. See the DD DIFFERENCES section
below. Default is 512 which to date has been correct for hard disks.
Other logical block sizes are 2048 bytes for DVDs and 4096 bytes for
the coming generation of hard disks.
cdbsz=6 | 10 | 12 | 16
size of SCSI READ and/or WRITE commands issued on pt devices.
Default is 10 byte SCSI command blocks (unless calculations indicate
that a 4 byte block number may be exceeded or BPT is greater than
16 bits (65535), in which case it defaults to 16 byte SCSI commands).
coe=0 | 1
set to 1 for continue on error. Applies to errors on input and output pt
devices plus input from block devices or regular files. Errors on other
files will stop ddpt. Default is 0 which implies stop on any error. See
the 'coe' flag for more information.
coe_limit=CL
where CL is the maximum number of consecutive bad blocks stepped
over (due to "coe=1") on reads before the copy terminates. The default
is 0 which is interpreted as no limit. This option is meant to stop the
copy soon after unrecorded media is detected while still
offering "continue on error" capability.
conv=CONVS
see the CONVERSIONS section below.
count=COUNT
copy COUNT input blocks from IFILE to OFILE. If this
option is not given (or COUNT is '-1') then the COUNT may be
deduced from either IFILE or OFILE. See the COUNT section below.
ibs=IBS
where IBS is the IFILE block size in bytes. The default value
is BS or its default (512). Conflicts the "bs=" option (e.g. giving
both "bs=512 ibs=512" is considered a syntax error).
if=IFILE
read from IFILE. This option must be given (i.e. it is mandatory). If
IFILE is '-' then stdin is read. Starts reading at the beginning of
IFILE unless SKIP is given.
iflag=FLAGS
where FLAGS is a comma separated list of one or more flags outlined
in the FLAGS section below. These flags are associated with IFILE and
are ignored when IFILE is stdin.
obs=OBS
where OBS is the OFILE block size in bytes. The default value
is BS or its default (512). Conflicts the "bs=" option (e.g. giving
both "bs=512 obs=512" is considered a syntax error).
If OBS is given then it has the following restriction: the integer
expression (((IBS * BPT) % OBS) == 0) must be true.
Stated another way: the copy buffer size must be an integral multiple of
OBS. If of2=OFILE2 is given then OBS is its block size
as well.
of=OFILE
write to OFILE. The default value is /dev/null . If OFILE is '-'
then writes to stdout. If OFILE is /dev/null then no actual writes are
performed. If OFILE is '.' (period) then it is treated the same way as
/dev/null . If OFILE exists then it is _not_ truncated
unless "oflag=trunc" is given. See section on DD DIFFERENCES.
of2=OFILE2
write output to OFILE2. The default action is not to do this additional
write (i.e. when this option is not given). OFILE2 is assumed to be
a regular file or a fifo (i.e. a named pipe). OFILE2 is opened for
writing and is created if necessary. If OFILE2 is a fifo (named pipe)
then some other command should be consuming that data (e.g. 'md5sum OFILE2'),
otherwise this utility will block. The write to OFILE2 occurs before
the write to OFILE and prior to sparse writing and write sparing
logic. So everything read is written to OFILE2.
oflag=FLAGS
where FLAGS is a comma separated list of one or more flags outlined
in the FLAGS section. These flags are associated with OFILE and are
ignored when OFILE is /dev/null, '.' (period), or stdout.
retries=RETR
sometimes retries at the host are useful, for example when there is a
transport error. When RETR is greater than zero then SCSI READs and
WRITEs are retried on error, RETR times. Default value is zero.
Only applies to errors on pt devices.
seek=SEEK
start writing SEEK blocks (each of OBS bytes) from the start of
OFILE. Default is block 0 (i.e. start of file). The SEEK value
may exceed the number of OBS-sized blocks in OFILE.
skip=SKIP
start reading SKIP blocks (each of IBS bytes) from the start of
IFILE. Default is block 0 (i.e. start of file). The SKIP value
must be less than or equal to the number of IBS-sized blocks in
IFILE.
status=STAT
the STAT value of 'noxfer' suppresses the throughput speed and the
copy time output at the end of the copy. The "status=noxfer" option was
recently introduced to GNU's dd command. The default action of ddpt is to
show the throughput (in megabytes per second) and the time taken to do the
copy after the "records in" and "records out" lines at the end of the copy.
As a convenience the value 'null' is accepted for STAT and does nothing.
verbose=VERB
as VERB increases so does the amount of debug output sent to stderr.
Default value is zero which yields the minimum amount of debug output.
A value of 1 reports extra information that is not repetitive. A value
2 reports cdbs and responses for SCSI commands that are not repetitive
(i.e. other that READ and WRITE). Error processing is not considered
repetitive. Values of 3 and 4 yield output for all SCSI commands, plus
Unix read() and write() calls, so there can be a lot of output.
If VERB is "-1" then output otherwise sent to stderr is redirected
to /dev/null .
-h, --help
outputs usage message and exits.
-v, --verbose
equivalent of verbose=1. If --verbose appears twice then
that is equivalent to verbose=2. Also -vv is equivalent to
verbose=2.
-V, --version
outputs version number information and exits.
-w, --wscan
this option is available in Windows only. It lists storage device names
and the corresponding volumes, if any. When used twice it adds the "bus
type" of the closest transport (e.g. a SATA disk in a USB connected
enclosure has bus type Usb). When used three times a SCSI adapter scan
is added. When used four times only a SCSI adapter scan is shown.
See EXAMPLES section below and the README.win32 file.
COUNT
When the count=COUNT option is not given (or COUNT is '-1')
then an attempt is made to deduce COUNT as follows.
When both or either IFILE and OFILE are block devices, then
the minimum size, expressed in units of input blocks, is used. When both
or either IFILE and OFILE are pass-through devices, then the
minimum size, expressed in units of input blocks, is used.
If a regular file is used as input, its size, expressed in units of input
blocks (and rounded up if necessary) is used. Note that the rounding up
of the deduced COUNT may result in a partial read of the last input
block and a corresponding partial write to OFILE if it is a regular
file.
The size of pt devices is deduced from the SCSI READ CAPACITY command.
Block device sizes (or their partition sizes) are obtained from the
operating system, if available.
If skip=SKIP or skip=SEEK are given and the COUNT is
deduced (i.e. not explicitly given) then that size is scaled back so
that the copy will not overrun the file or device.
If COUNT is not given and IFILE is a fifo (and stdin is
treated as a fifo) then IFILE is read until an EOF is detected.
If COUNT is not given and IFILE is a /dev/zero (or
equivalent) then zeros are read until an error occurs (e.g. file
system full).
If COUNT is not given and cannot be deduced then an error message
is issued and no copy takes place.
CONVERSIONS
One or more conversions can be given to the "conv=" option. If more than
one is given, they should be comma separated. ddpt does not perform the
traditional dd conversions (e.g. ASCII to EBCDIC). Recently added
conversions overlap somewhat with the flags so some conversions are
now supported by ddpt.
fdatasync
equivalent to "oflag=fdatasync". Flushes data associated with the
OFILE to storage at the end of the copy. This conversion is
for compatibility with GNU's dd.
fsync
equivalent to "oflag=fsync". Flushes data and meta-data associated
with the OFILE to storage at the end of the copy. This conversion
is for compatibility with GNU's dd.
noerror
this conversion is very close to "iflag=coe" and is treated as such. See
the "coe" flag. Note that an error on OFILE will stop the copy.
null
has no affect, just a placeholder.
resume
See "resume" in the FLAGS sections for more information.
sparing
See "sparing" in the FLAGS sections for more information.
sparse
FreeBSD supports "conv=sparse" so the same syntax is supported in ddpt.
See "sparse" in the FLAGS sections for more information.
sync
is ignored by ddpt. With dd it means supply zero fill (rather than skip)
and is typically used like this "conv=noerror,sync" to have the same
functionality as ddpt's "iflag=coe".
trunc
if OFILE is a regular file then truncate it prior to starting the
copy. See "trunc" in the FLAGS section.
FLAGS
A list of flags and their meanings follow. The flag name is followed
by one or two indications in square brackets. The first indication is
either "[i]", "[o]" or "[io]" indicating this flag is active for the
IFILE, OFILE or both the IFILE and the OFILE. The
second indication contains some combination of "reg", "blk" or "pt"
indicating whether the flag applies to a regular file, a block
device (accessed via Unix read() and write() commands) or a pass-through
device respectively.
append [o] [reg]
causes the O_APPEND flag to be added to the open of OFILE. For
regular files this will lead to data appended to the end of any existing
data. Conflicts the seek=SEEK option. The default action of this
utility is to overwrite any existing data from the beginning of OFILE
or, if SEEK is given, starting at block SEEK. Note that
attempting to 'append' to a device file (e.g. a disk) will usually be
ignored or may cause an error to be reported.
coe [io] [pt], [i] [reg,blk]
continue on error. 'iflag=coe oflag=coe' and 'coe=1' are equivalent.
Errors occurring on output regular or block files will stop ddpt.
Error messages are sent to stderr. This flag is similar
to 'conv=noerror,sync' in the
dd(1)
utility. Unrecovered errors are counted and output in the summary at
the end of the copy.
This paragraph is about coe on pt devices. A
medium, hardware or blank check error while reading will re-read blocks
prior to the bad block, then try to recover the bad block, supplying zeros
if that fails, and finally reread the blocks after the bad block. A
medium, hardware or blank check error while writing is noted and ignored.
SCSI disks may automatically try and remap faulty sectors (see the AWRE
and ARRE in the read write error recovery mode page (the sdparm utility
can access these attributes)). If bad LBAs are reported by the
pass-through then the LBA of the lowest and highest bad block is also
output.
This paragraph is about coe on input regular files and block devices.
When a EIO or EREMOTEIO error is detected on a normal segment read then
the segment is re-read one block (i.e. IBS bytes) at a time. Any
block that yields a EIO or EREMOTEIO error is replaced by zeros. Any
other error, a short read or an end of file will terminate the copy,
usually after the data that has been read is written to the output file.
direct [io] [reg,blk]
causes the O_DIRECT flag to be added to the open of IFILE and/or
OFILE. This flag requires some memory alignment on IO. Hence user
memory buffers are aligned to the page size. May have no effect on pt
devices. This flag will bypass caching/buffering normally done by block
layer. Beware of data coherency issues if the same locations have been
recently accessed via the block layer in its normal mode (i.e.
non-direct). See open(2) man page.
dpo [io] [pt]
set the DPO bit (disable page out) in SCSI READ and WRITE commands. Not
supported for 6 byte cdb variants of READ and WRITE. Indicates that
data is unlikely to be required to stay in device (e.g. disk) cache.
May speed media copy and/or cause a media copy to have less impact
on other device users.
errblk [i] [pt] [experimental]
attempts to create or append to a file called "errblk.txt" in the current
directory the logical block addresses of blocks that cannot be read. The
first (appended) line is "# start <timestamp>". That is followed by the
LBAs in hex (and prefixed with "0x") of any block that cannot be read,
one LBA per line. If the sense data does not correctly identify the LBA of
the first error in the range it was asked to read then a LBA range is
output in the form of the lowest and the highest LBA in the range
separated by a "-". At the end of the copy a line with "# stop <timestamp>"
is appended to "errblk.txt". Typically used with "coe".
excl [io] [reg,blk]
causes the O_EXCL flag to be added to the open of IFILE and/or
OFILE. See open(2) man page.
fdatasync [o] [reg,blk]
Flushes data associated with the OFILE to storage at the end of the
copy.
flock [io] [reg,blk,pt]
after opening the associated file (i.e. IFILE and/or OFILE)
an attempt is made to get an advisory exclusive lock with the flock()
system call. The flock arguments are "FLOCK_EX | FLOCK_NB" which will
cause the lock to be taken if available else a "temporarily unavailable"
error is generated. An exit status of 90 is produced in the latter case
and no copy is done. See flock(2) man page.
force [io] [pt]
override difference between given block size and the block size found
by the SCSI READ CAPACITY command. Use the given block size. Without
this flag the copy would not be performed. pt access to what appears
to be a block partition is aborted in version 0.92; that can be overridden
by the force flag. For related reasons the 'norcap' flag requires this
flag when applied to a block device accessed via pt.
fsync [o] [reg,blk]
Flushes data and metadata (describing the file) associated with the
OFILE to storage at the end of the copy.
fua [io] [pt]
causes the FUA (force unit access) bit to be set in SCSI READ and/or WRITE
commands. The 6 byte variants of the SCSI READ and WRITE commands do not
support the FUA bit.
fua_nv [io] [pt]
causes the FUA_NV (force unit access non-volatile cache) bit to be set in
SCSI READ and/or WRITE commands. This only has an effect with pt devices.
The 6 byte variants of the SCSI READ and WRITE commands do not support the
FUA_NV bit.
nocache [io] [reg,blk]
use posix_fadvise() to advise corresponding file there is no need to fill
the file buffer with recently read or written blocks. If used with "iflag="
it will increase the read ahead on IFILE.
norcap [io] [pt]
do not perform SCSI READ CAPACITY command on the corresponding pt device.
If used on block device accessed via pt then 'force' flag is also
required. This is to warn about using pt access on what may be a block
device partition.
nowrite [o] [reg,blk,pt]
bypass writes to OFILE. The "records out" count is not incremented.
OFILE is still opened but "oflag=trunc" if given is ignored. Also
the ftruncate call associated with the sparse flag is ignored (i.e.
bypassed). Commands such as trim and SCSI SYNCHRONIZE CACHE are still sent.
null [io]
has no affect, just a placeholder.
pt [io] [blk,pt]
causes a device to be accessed in "pt" mode. In "pt" mode SCSI READ and
WRITE commands are sent to access blocks rather than standard UNIX read()
and write() commands. The "pt" mode may be implicit if the device is only
capable of passing through SCSI commands (e.g. the /dev/sg devices in
Linux). This flag is needed for device nodes that can be accessed both
via standard UNIX read() and write() commands as well as SCSI commands.
Such devices default standard UNIX read() and write() commands in the
absence of this flag.
resume [o] [reg]
when a copy is interrupted (e.g. with Control-C from the keyboard)
then using the same invocation again with the addition of "oflag=resume"
will attempt to restart the copy from the point of the interrupt (or
just before that point). It is harmless to use "oflag=resume" when
OFILE doesn't exist or is zero length. If the length of OFILE
is greater than or equal to the length implied by a ddpt invocation that
includes "oflag=resume" then no further data is copied.
self [io] [pt]
used together with trim flag to do a self trim (trim of segments of a
pt device that contain all zeros). If OFILE is not given, then
it is set to the same as IFILE. If SEEK is not given it
set to the same value as SKIP (possibly adjusted if IBS
and OBS are different). Implicitly sets "nowrite" flag.
sparing [o] [reg,blk,pt]
during the copy each IBS * BPT byte segment is read from
IFILE into a buffer. Then, instead of writing that buffer to
OFILE, the corresponding segment is read from OFILE into another
buffer. If the two buffers are different, the former buffer is written to
the OFILE. If the two buffers compare equal then the write to
OFILE is not performed. Write sparing is useful when a write operation
is significantly slower than a read. Under some conditions flash memory
devices have slow writes plus an upper limit on the number of times the same
cell can be rewritten. The granularity of the comparison can be reduced from
the default IBS * BPT byte segment with the the OBPC value
given to the "bpt=" option. The finest granularity is when OBPC is 1
which implies OBS bytes.
sparse [o] [reg,blk,pt]
after each IBS * BPT byte segment is read from IFILE, it
is checked to see if it is all zeros. If so, that segment is not written to
OFILE. See the section on SPARSE WRITES below. The granularity of
the zero comparison can be reduced from the default IBS * BPT
byte segment with the OBPC value given to the "bpt=" option.
ssync [o] [pt]
if OFILE is in "pt" mode then the SCSI SYNCHRONIZE CACHE command is
sent to OFILE at the end of the copy.
strunc [o] [reg]
perform a sparse copy with a ftruncate system call to extend the length
of the OFILE if required. See the sparse flag and the section on
SPARSE WRITES below.
sync [io] [reg,blk]
causes the O_SYNC flag to be added to the open of IFILE and/or
OFILE. See open(2) man page.
trim [io] [pt] [experimental]
similar logic to the "sparse" option. However instead of skipping segments
that are full of zeros a "trim" command is sent to OFILE. Usually set
as an oflag argument but for self trim can be used as an iflag
argument (e.g. "iflag=self,trim"). Depending on the usage this may require
the device to support "deterministic read zero after trim". See the
TRIM, UNMAP AND WRITE SAME section below.
trunc [o] [reg]
if OFILE is a regular file then it is truncated prior to starting the
copy. If SEEK is not given or 0 then OFILE is truncated to zero
length; when SEEK is larger than zero the truncation takes place at
file byte pointer SEEK*OBS. Ignored if "oflag=append". Conflicts
with "oflag=sparing".
SPARSE WRITES
Bypassing writes of blocks full of zeros can save a lot of IO. However
with regular files, bypassed writes at the end of the copy can lead
to an OFILE which is shorter than it would have been without
sparse writes. This can lead to integrity checking programs like md5sum
and sha1sum generating different values.
This utility has two ways of handling this file length problem: writing
the last block (even if it is full of zeros) or using the ftruncate
system call. A third approach is to ignore the problem (i.e. leaving
OFILE shorter). The ftruncate approach is used when "oflag=strunc"
while the last block is written when "oflag=sparse". To ignore the
file length issue use "oflag=sparse,sparse". Note that if OFILE's
length is already correct or longer than required, no action is taken.
The support for sparse writing of regular files may depend on the OS, the
file system and the settings of OFILE. POSIX makes few guarantees
when the ftruncate system call is used to extend a file's length, as may
occur when "oflag=strunc". Further, primitive file systems like VFAT may not
accept sparse writes or simulate the effect by writing blocks of zeros. The
latter approach will defeat any sparse writing performance gain.
TRIM, UNMAP AND WRITE SAME
This is a new storage feature often associated with Solid State
Disks (SSDs) or disk arrays with "thin provisioning". In the ATA command
set (ACS-2) the relevant command is DATA SET MANAGEMENT with the TRIM
bit set. In the SCSI command set (SBC-3) it is either the UNMAP or
WRITE SAME command. Note there is no TRIM command however the term is
frequently used in the technical press.
Trim is a way of telling a storage device that blocks are no longer needed.
Keeping the pool of unwritten blocks large is important for the write
performance of SSDs and the thrifty use of real storage in thin provisioned
arrays. Currently file systems in recent OSes may issue trims associated
with file deletes. The trim option in ddpt may be useful when a partition
or a whole SSD is to be "deleted". Note that ddpt is bypassing file
systems in that it only offers trim on pass-through (pt) devices.
This utility issues SCSI commands to pt devices and for "trim" currently
issues a SCSI WRITE SAME(16) command with the UNMAP bit set. If the pt
device is a SSD with a ATA interface then recent versions of Linux
will translate the SCSI WRITE SAME to the ATA DATA SET MANAGEMENT command
with the TRIM bit set. The maximum size of each "trim" command sent
is the size of the copy buffer (i.e. IBS * BPT bytes). And
that maximum can be reduced with the OBPC argument of the "bpt="
option.
The trim can be used various ways. One way is a copy where the copy
buffer (or some part of it) is checked for zeros as is done by the
sparse oflag. When a zero segment is found, a trim "command" is
sent to the OFILE. For example:
ddpt if=dsk.img bs=512 of=/dev/sdc oflag=pt,trim
The copy buffer is 64 KiB (since BPT and OBPC default to 128
when "bs=512") and it is checked for all zeros. If it is all zeros then
a trim command is sent to the corresponding location of /dev/sdc
which is accessed via the pt interface. If it is not all zeros
then a SCSI WRITE command is sent. Another way is to trim all or
part of a disk. To trim a whole disk (i.e. deleting all its data):
The "self" oflag automatically sets up the output side of the copy
to send trim commands (if required) back the the same device (i.e. /dev/sdc).
If this example was self-trimming a partition then the partition would
start at LBA 0x2300 and be 0x1234f0 blocks long.
Some random product examples: the Intel X25-M G2 SSDs have trim with
recent firmware and they do deterministic read zero after trim. The
Seagate Pulsar SSD has an ATA interface which supports the deterministic
reads of zero after the DATA SET MANAGEMENT command with the TRIM option.
DD DIFFERENCES
dd defaults "if=" and "of=" to stdin and stdout respectively. This follows
Unix filter conventions. However since dd and ddpt are often used to read
binary data for timing purposes, having to supply "of=/dev/null" can
be easily forgotten. Without it dd will potentially spew binary data on the
console. So ddpt has changed its defaults: the "if=IFILE" is now
mandatory and to read from stdin "if=-" can be used; "of=OFILE"
remains optional but its default changes to "/dev/null" (or "NUL" in
Windows). To send output to stdout ddpt accepts "of=-".
dd truncates OFILE unless "conv=notrunc" is given. When dd truncates,
it truncates to zero length unless SEEK is greater than zero. ddpt
does not truncate OFILE by default. If OFILE exists it will be
overwritten. The overwrite starts at block zero unless SEEK
or "oflag=append" is given. If OFILE is a regular file
then "oflag=trunc" (or "conv=trunc") will truncate OFILE prior to the
copy.
Numeric arguments to ddpt can be given in hexadecimal, either with a
leading "0x" or "0X" or with a trailing "h". Note that dd accepts "0x123"
but interprets it as "0 * 123" (i.e. zero). ddpt will also interpret "x"
as multiplies unless the left operand is zero (e.g. "0x123"). So both
dd and ddpt will interpret "skip=2x123" as "skip=246".
Terabyte size disks make it impractical to copy all the data into a buffer
before writing it out. Therefore both dd and ddpt read a relatively small
amount of data into a copy (or transfer) buffer then write it out to the
destination, repeating this process until the COUNT is exhausted.
A major difference in ddpt is the addition of BPT to control the
size of the copy buffer. With dd, IBS is the size of the copy buffer
and the unit of SKIP and COUNT. With ddpt, IBS * BPT
is the size of the copy buffer and IBS is the unit of SKIP
and COUNT. This allows ddpt to have its IBS set to the logical
block size of IFILE without unduly restricting the size of the copy
buffer. And setting IBS (and OBS for OFILE) accurately
is required when the pass-through interface is used since with the SCSI
READ and WRITE commands the logical block size is implicit.
The way dd handles its copy buffer (outlined in SUSv4 description of dd)
is relatively complex, especially when IBS and OBS are different
sizes. The restriction that ddpt places on IBS and OBS (
i.e. (((IBS*BPT) % OBS) == 0) ) means that a single
copy buffer can be used since its size is a multiple of both IBS and
OBS. Being able to precisely define the copy buffer size in ddpt
makes sparse writing, write sparing and trim operations simpler to
define and the user to control.
ddpt does not support dd's "cbs=" option (conversion block size). If
the "cbs=" option is given to ddpt then it is ignored.
NOTES
A partial write is a write to the OFILE of less than OBS
bytes. This typically occurs at the end of a copy. dd can do partial
writes. ddpt does partial writes to regular files and fifos (including
stdout). However ddpt ignores partial writes when OFILE is a block
device or a pt device. When ddpt ignores a partial write, it sends a
warning to the console (stderr).
At the end of the copy two lines are output to the console:
<in_full>+<in_partial> records in
<out_full>+<out_partial> records out
The "records in" line is the number of full input blocks (each of
IBS bytes) that have been read plus the number of partial blocks (
usually less than IBS bytes) that have been read. Following the lead
of dd when 'iflag=coe' is active a block that cannot be read (and has zeros
substituted for its output) is regarded as a partial read. The "records out"
line is the number of full output blocks (each of OBS bytes) that
have been written plus the number of partial blocks (usually less than
OBS bytes) that have been written.
Block devices (e.g. /dev/sda and /dev/hda) can be given for IFILE.
If neither 'iflag=direct' nor 'iflag=pt' is given then normal block IO
involving buffering and caching is performed. If 'iflag=direct' is given
then the buffering and caching is bypassed (this is applicable to both SCSI
devices and ATA disks). When 'iflag=pt' is given SCSI commands are sent to
the device which bypasses most of the actions performed by the block layer.
The same applies for block devices given for OFILE.
BPT, BS, COUNT, IBS, OBPC, OBS,
SKIP and SEEK may include one of these multiplicative suffixes:
c C *1; w W *2; b B *512; k K KiB *1,024; KB *1,000; m M MiB *1,048,576;
MB *1,000,000 . This pattern continues for "G", "T" and "P". The latter two
suffixes can only be used for COUNT, SKIP and SEEK.
Also a suffix of the form "x<n>" multiplies the leading number by <n>;
however the combinations "0x" and "0X" are treated differently, see the
next paragraph. These multiplicative suffixes are compatible with GNU's
dd command (since 2002) which claims compliance with the SI and with
IEC 60027-2 standards.
Alternatively numerical values can be given in hexadecimal preceded by
either "0x" or "0X" (or with a trailing "h" or "H"). When hex numbers are
given, multipliers cannot be used.
The COUNT, SKIP and SEEK arguments can take 64 bit
values (i.e. very big numbers). Other numerical values are limited to what
can fit in a signed 32 bit number.
All informative, warning and error output is sent to stderr so that
dd's output file can be stdout and remain unpolluted. If no options
are given, then the usage message is output and nothing else happens.
Disk partition information can often be found with
fdisk(8)
[the "-ul" argument is useful in this respect]. Also
parted(8)
can be used like this: 'parted /dev/sda unit s print' .
For pt devices this utility issues SCSI READ and WRITE (SBC) commands which
are appropriate for disks and reading from CD/DVD/BD drives. Those
commands are not formatted correctly for tape devices so ddpt should not be
used on tape devices. If the largest block address of the requested transfer
exceeds a 32 bit block number (i.e 0xffffffff) then a warning is issued and
the sg device is accessed via SCSI READ(16) and WRITE(16) commands.
The attributes of a block device (e.g. partitions) are ignored when thept flag is used.
Hence the whole device is read (rather than just the second partition) by
this invocation:
ddpt if=/dev/sdb2 iflag=pt of=t bs=512
Assuming /dev/sdb and /dev/sg2 refer to the same device, then after the
following two invocations, the contents of the files "t", "tt" and "ttt"
should be same:
ddpt if=/dev/sdb of=tt bs=512
ddpt if=/dev/sg2 of=ttt bs=512
EXAMPLES
The examples in this page use Linux device names. For suitable device
names in other supported Operating Systems see this web page:
http://sg.danny.cz/sg/device_name.html . The sg3_utils(8) man page
in the sg3_utils package also covers device naming.
ddpt usage looks quite similar to dd:
ddpt if=/dev/sg0 of=t bs=512 count=1MB
This will copy 1 million 512 byte blocks from the device associated with
/dev/sg0 (which should have 512 byte blocks) to a file called t.
Assuming /dev/sda and /dev/sg0 are the same device then the above is
equivalent to:
although dd's speed may improve if bs was larger and count was suitably
reduced. The use of the 'iflag=direct' option bypasses the buffering and
caching that is usually done on a block device.
The dd command's bs argument can be thought of as roughly equivalent to
ddpt's bs*bpt . dd almost assumes buffering on a block device and will
work as long as bs is a multiple of the actual logical block size.
Since ddpt can work at a lower level in some cases the bs argument must be
a disk's actual logical block size. Thus the bpt argument was introduced
to make the copy more efficient. So these two invocations are roughly
equivalent:
dd if=/dev/sda of=t bs=8k count=64
ddpt if=/dev/sda of=t bs=512 bpt=16 count=1k
In both cases the total number of bytes moved is bs*count . And that will
be done by reading 8k (8192 bytes) into a buffer then writing out that
buffer to the file t. The read write sequence continues until the
count is complete or an error occurs.
The 'of2=' option can save time when the input would otherwise need to be
read twice. For example, to copy data and take a md5sum of it without
needing to re-read the data:
This will image /dev/sg3 (e.g. an unmounted disk) and place the contents
in the (sparse) file sg3.img . Without re-reading the data it will also
perform a md5sum calculation on the image.
Now we use sparse writing logic to get some idea of how many blocks
on a disk are full of zeros. After a SCSI FORMAT or an ATA SECURITY ERASE
command a disk may be all zeros.
ddpt if=/dev/sdc bs=512 oflag=sparse
Since no "of=" option is given, output goes to /dev/null so nothing
is actually written so the "records out" will be zero. However there
will be a count of "records in" and "bypassed records out". If /dev/sdc is
full of zeros then "records in" and "bypassed records out" will be
the same. Since the "bpt=" option is not given it defaults to "bpt=128,128"
so the copy buffer will be 64 KiB and the sparse check for zeros will
be done with 64 KiB (128 block) granularity.
For examples of the trim and self,trim options see the section above
on TRIM, UNMAP AND WRITE SAME.
Following is an example run on a Windows OS using the '--wscan' option
which shows the available device names (e.g. PD1) and the associated volume
name(s):
So, for example, volumes D: and F: reside on PhysicalDisk1 (abbreviated to
"PD1") which is manufactured by WD (Western Digital).
Further examples can be found on this web page:
http://sg.danny.cz/sg/ddpt.html .
There is a text file called ddpt_examples.txt in the "doc" directory of
this package's distribution tarball.
SIGNALS
The signal handling has been borrowed from dd: SIGINT, SIGQUIT and
SIGPIPE output the number of remaining blocks to be transferred and
the records in + out counts; then they have their default action.
SIGUSR1 causes the same information to be output and the copy continues.
All output caused by signals is sent to stderr.
EXIT STATUS
To aid scripts that call ddpt, the exit status is set to indicate
success (0) or failure (1 or more). Note that some of the lower values
correspond to the SCSI sense key values. The exit status values are:
0
success
1
syntax error. Either illegal command line options, options with bad
arguments or a combination of options that is not permitted.
2
the device reports that it is not ready for the operation requested.
The device may be in the process of becoming ready (e.g. spinning up but
not at speed) so the utility may work after a wait.
3
the device reports a medium or hardware error (or a blank check). For example
an attempt to read a corrupted block on a disk will yield this value.
5
the device reports an "illegal request" with an additional sense code other
than "invalid operation code". This is often a supported command with a
field set requesting an unsupported capability.
6
the device reports a "unit attention" condition. This usually indicates
that something unrelated to the requested command has occurred (e.g. a
device reset) potentially before the current SCSI command was sent. The
requested command has not been executed by the device. Note that unit
attention conditions are usually only reported once by a device.
9
the device reports an illegal request with an additional sense code
of "invalid operation code" which means that it doesn't support the
requested command.
11
the device reports an aborted command. In some cases aborted commands can
be retried immediately (e.g. if the transport aborted the command due to
congestion).
15
the utility is unable to open, close or use the given IFILE or
OFILE. The given file name could be incorrect or there may be
permission problems. Adding the -v option may give more information.
20
the device reports it has a check condition but "no sense".
It is unlikely that this value will occur as an exit status.
21
the device reports a "recovered error". The requested command was successful.
Most likely a utility will report a recovered error to stderr and continue,
probably leaving the utility with an exit status of 0 .
33
the command sent to device has timed out. This occurs in Linux only; in
other ports a command timeout will appear as a transport (or OS) error.
90
the flock flag has been given on a device and some other process holds the
advisory exclusive lock.
97
the response to a SCSI command failed sanity checks.
98
the device reports it has a check condition but the error doesn't fit into
any of the above categories.
99
any errors that can't be categorized into values 1 to 98 may yield
this value. This includes transport and operating system errors
after the command has been sent to the device.