[clue-tech] How does bytesize affect dd?

[Jim Ockers]

Hi,

Do you know why your mailer is sending out multipart/mixed e-mails
instead of plain text?  There's a bunch of garbage that's hard to
parse and then the HTML that I had to delete.

Hani Duwaik wrote:
> 
> --===============1809396246==
> Content-Type: multipart/alternative; 
> 	boundary="----=_Part_15777_29443526.1135118313437"
> 
> ------=_Part_15777_29443526.1135118313437
> Content-Type: text/plain; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> Content-Disposition: inline
> 
> I was wondering if anyone could shed some light on the 'bs' option of the
> 'dd' command...
> 
> Specifically, I know that in the following command:
> 
> dd if=3Dxxx of=3Dyyy bs=3Dzzz
> 
> will copy the contents of 'xxx' to 'yyy' block-by-block with a block size o=
> f
> 'zzz' ... however, my question is what affect increasing 'bs' would have
> aside from possibly speeding things up.

I would think that bs is the number of bytes passed as an argument to
write(2).  From the man page for write(), the third argument is the
number of bytes to write in that one system call.  I'm assuming that
dd uses write(2) but that seems pretty likely to me.

The only time I've ever been interested in the chunksize is when I was
doing filesystem benchmarking on IDE compactflash and I'm pretty sure
I exposed a timing problem (heisenbug) with the following 3 conditions:

1. IDE compactflash other than Sandisk ExtremeIII
2. ext3 filesystem formatted with 1024 or 4096 byte blocksize, mounted
with noatime option
3. large chunksizes written by write(2)

The failure manifests itself as an ENOSPACE or EBADF (no such file or
directory) returned by the filesystem to errno, and write(2) returns -1 .
I was using bonnie++ for benchmarking, and bonnie++ accepts a command
line argument for chunksize, just like dd.  The default chunksize is 8K
which is "large."

If you use very small chunksizes (less than 256 bytes), then the write(2)
operation is slowed down enough that everything works fine.  This is
consistent with your conclusion that large blocksizes would in general
speed things up.

If you use ext2 it's also fine.  Or, if you put printk() debugging
messages in the ext3 code to see what is going on, it slows things 
down enough that everything works fine.  (heisenbug!)

If you use the Sandisk ExtremeIII it's also fine.  We think this is
because the ExtremeIII either has a large cache or is extremely fast
compared to all the other IDE compactflash we tried.  Yes we tried
several of each variety and vendor that we could get.

This is all very repeatable - we also tried different IDE chipsets and
thus kernel IDE drivers.  DMA didn't seem to make too much of a diff-
erence if on or off.

The ext3 filesystem blocksize can be 512, 1024, or 4096.  It didn't
seem to make any difference - if the write() chunksize was >256 we
got errors, otherwise we didn't.

> In addition, how do the conv flags "noerror, notrunc, sync" come into play.

We put a fsync() in the bonnie++ code right after the write() to see 
what would happen if the filesystem was synced after every write(),
but that slowed it down so much that it was unusable.  (And of course
there were no errors.)

I'm not sure what you mean by conv flags but I would guess that the 
sync option probably tries to do a fsync() periodically, maybe after 
every write().

> I'm particularly interested in how this applies to reading/writing CDs/DVDs=
> .

Not sure about that, but some devices or filesystems probably have 
a preferred block size.

Hope this helps, if not to solve your problem to at least understand
what's going on...

Jim

> 
> TIA,
> 
> -Hani
> 
> --
> "If, of the many truths, you select only one and follow it blindly, it will
> become a falsehood, and you a fanatic."a
> 
> ------=_Part_15777_29443526.1135118313437
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> Content-Disposition: inline
> 
> I was wondering if anyone could shed some light on the 'bs' option of the '=
> dd' command...<br>
> <br>
> Specifically, I know that in the following command:<br>
> <br>
> dd if=3Dxxx of=3Dyyy bs=3Dzzz<br>
> <br>
> will copy the contents of 'xxx' to 'yyy' block-by-block with a block
> size of 'zzz' ... however, my question is what affect increasing 'bs'
> would have aside from possibly speeding things up.<br>
> <br>
> In addition, how do the conv flags &quot;noerror, notrunc, sync&quot; come =
> into play.<br>
> <br>
> I'm particularly interested in how this applies to reading/writing CDs/DVDs=
> .<br>
> <br>
> TIA,<br>
> <br>
> -Hani<br clear=3D"all"><br>-- <br>&quot;If, of the many truths, you select =
> only one and follow it blindly, it will become a falsehood, and you a fanat=
> ic.&quot;a
> 
> ------=_Part_15777_29443526.1135118313437--
> 
> --===============1809396246==
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
> 
> _______________________________________________
> CLUE-tech mailing list
> CLUE-tech at cluedenver.org
> http://cluedenver.org/mailman/listinfo/clue-tech
> --===============1809396246==--
> 


_______________________________________________
CLUE-tech mailing list
CLUE-tech at cluedenver.org
http://cluedenver.org/mailman/listinfo/clue-tech