| Xenguy | rwp: Well spoken | 07:03 |
|---|---|---|
| Xenguy | Or one can simply learn to use 'find' as it is | 07:03 |
| Xenguy | http://mywiki.wooledge.org/UsingFind | 07:08 |
| FatPhil | I think dd takes the prize for not fitting in with the rest of the unix programs. | 10:31 |
| hagbard | And I'm never all to sure of which block size would be the optimum for speed. | 10:45 |
| al1r4d | Assalamualaikum :) | 16:17 |
| fsmithred | radio1059 | 16:29 |
| debdog | fsmithred_: in Valparaiso? | 16:33 |
| debdog | search was inconclusive | 16:33 |
| fsmithred_ | oh, I'll have to log out and in again. I've got shitty satellite internet that says I'm somewhere about 800 miles away from where I really am. | 16:36 |
| fsmithred_ | I actually have been to valparaiso, but that was 40 years ago and only for a day. | 16:37 |
| Xenguy | hagbard: In my notes from 5 years ago I have: dd bs=4M | 17:28 |
| Xenguy | etc. Supposed to be faster, not that that is terribly important really | 17:28 |
| debdog | when going for faster, use dd -if | dd -of | 17:35 |
| debdog | so much for satellite internet | 22:09 |
| TorC | Xenguy, hagbard: IME, using BS=8M to 16M definitely speeds up transfers to high speed USB sticks speeds things up quite noticeably. To the point I'll sometimes cancel and restart if I realize I missed it. | 22:27 |
| hagbard | Thats what i always use too. | 22:27 |
| TorC | I haven't noticed any visible difference between the two, however, and bs=4M is likely to do just fine, too. | 22:28 |
| fluffywolf | the only times I've noticed a difference with >1M is writing to shingled drives. | 22:35 |
| bb|hcb | TorC: Isn't 1M just enough? | 22:37 |
| rwp | I avoid SMR drives as not suitable for (my) purpose. I always use 16M too. As long as it is equal to, or an integer multiple of, the system block size then it's good. | 22:38 |
| rwp | Though my favorite USB storage writing command I use 1M for the progress bar accuracy: pv foo.iso | dd of=/dev/sdX bs=1M oflag=sync,direct | 22:39 |
| rwp | Then pv will display a close to mostly accurate progress bar of the write progress. | 22:39 |
| Xenguy | Nice, I ran across pv at some point, but never bothered to try it out | 22:43 |
| bb|hcb | pv is a life saver, esp. when you "ssh host tar cf - bla/|pv|tar fx -" | 22:52 |
| Xenguy | I wonder what 'pv' stands for? 'Progress ___________' ? | 23:00 |
| rwp | "pv" == "pipe viewer" | 23:02 |
| rwp | http://www.ivarch.com/programs/pv.shtml | 23:02 |
| rwp | It's one of those utilities that I admit I initially discounted but now it is one that I can't be without! | 23:02 |
| TorC | bb|hcb: 1M may well be enough. I first started with something larger, it works, so I haven't bothered to test various permutations. | 23:09 |
| gnarface | it all depends on the speed of what you're writing to | 23:11 |
| gnarface | for most cheap USB/SD flash devices, the default is about as fast as it can go already, increasing it usually does nothing but can actually slow it down, or in really rare cases even corrupt the write | 23:12 |
| gnarface | you need something really fast like NVME/SSD/eMMC media to be able to actually benefit from high block size writes | 23:12 |
| gnarface | though, some more high quality USB or SD flash devices have a decent amount of high-speed cache, which can fool you for small data sets | 23:13 |
| gnarface | but once you exhaust the cache, write speed will plummet back down to normal speeds | 23:14 |
| gnarface | (which, even for the best quality of such media, is still around 1/3 or less the speed of spinning rust) | 23:15 |
| TorC | Interesting. About the smallest transfer I deal with is 300-400MB, since I pretty much only use it for OS install disks, and even netinstalls have grown to about that size. | 23:16 |
| TorC | I would've expected that would be past where any cache effect would likely be visible. | 23:17 |
| gnarface | hmm, i would think so too but i can't be sure | 23:19 |
| gnarface | oh, but depends on what you're writing through too | 23:20 |
| TorC | Ah, right. I guess that means lower cost high speed writers are often tuned for photographers, and therefore expect a typical file size of 8- | 23:21 |
| TorC | 24MB, which would affect their handling of small blocks. | 23:21 |
| gnarface | yea, for example; 512MB raspberry pi model B, running an old kernel with a small memory footprint - due to the hardware layout and the way the SoC is connected to the SD card slot, you'll still get most of the the RPI's 512MB of on-chip system ram effectively also behaving as a buffer cache for writes to the SD from over the network in this situation | 23:21 |
| gnarface | with dd, the source system's kernel write-back cache mechanism can also obscure performance bottlenecks, unless you use "conv=fsync" on the dd command-line | 23:24 |
| gnarface | (or, alternately you can just run "sync" as a second step and count the time it takes to finish to gauge how much of the write speed is "cheats") | 23:25 |
| gnarface | dd is a special case, but in the RPI situation, you'll even see that behavior just downloading stuff over http from inside the running system | 23:26 |
| TorC | Good to know. Maybe I'll think of playing with it a bit sometime to see. | 23:26 |
| gnarface | ... 512MB of "woohoo!" speed then BAM, performance hits a wall | 23:26 |
| TorC | I know I see the effect with more normal transfers, where the time it takes for umount to do its job can run long. Hadn't known dd can also be subject to it. | 23:27 |
| gnarface | dd didn't always used to be that way, there was i think a kernel change around a decade or so back | 23:28 |
| gnarface | so now write-back cache can fool dd | 23:28 |
| gnarface | (i think it was an unintended consequence of some general USB block device optimizations, but the details are above my head) | 23:29 |
| bb|hcb | dd has oflag=sync to avoid this | 23:29 |
| TorC | That's good to be aware of, as I was under the impression as soon as DD finished I could safely remove the disk. No umount command needing to be run to prove it's done, either. | 23:29 |
| AzumaHazuki | i run with conv=sync,noerror | 23:29 |
| gnarface | bb|hcb: NOTE: you should use conv=fsync, because sync will only force the meta-data to sync | 23:29 |
| AzumaHazuki | ....oh god, really? D: i had no idea | 23:30 |
| gnarface | TorC: yea, that as a safe assumption with dd sometime before 2012 | 23:30 |
| gnarface | was* | 23:30 |
| gnarface | not sure quite how long before though | 23:31 |
| bb|hcb | What metadata?! Aren't we wring to a block device? :) If you dd to a file on a filesystem, then yes | 23:31 |
| gnarface | bb|hcb: i dunno, just what the man page says | 23:31 |
| gnarface | it might not matter in practice, but i think i had a problem with it just once... | 23:32 |
| gnarface | i'm sure there's a distinction for a reason anyway | 23:32 |
| gnarface | hmm, although i note that it seems oflag=sync and conv=sync aren't the same thing | 23:34 |
| gnarface | i might be thinking of conv=sync vs conv=fsync | 23:34 |
| gnarface | according to the man page, oflag=sync does... something else entirely | 23:34 |
| gnarface | something about using space instead of null | 23:35 |
| gnarface | maybe that'll somehow force the same effect, i dunno | 23:36 |
| gnarface | hmm, there's some new options i don't remember here before now, like dsync | 23:37 |
| bb|hcb | Finger memory is hard to change, the safer bet for all cases should be conv=fsync though; I am guessing that conv=fsync on a block device is doing the same thing as oflag=sync | 23:37 |
| rwp | Beware of conv=sync as that does padding on short reads! | 23:38 |
| bb|hcb | Yes, conv=sync may produce very unexpected results | 23:38 |
| gnarface | it occurs to me that at some point i was writing bash scripts to use a subset of command-line options to be compatible with bsd, linux and osx versions of some tools | 23:39 |
| gnarface | dd was one of them | 23:39 |
| gnarface | (ostensibly the osx versions were all supposed to be the same as the GNU versions despite it having kernel that came from bsd, but we found that they'd sometimes fork it then patch it up with weird shit) | 23:41 |
| rwp | It's unfortunate that OpenBSD's open(2) call does not support O_DIRECT which then breaks that otherwise common action from being portable to OpenBSD. | 23:48 |
| rwp | All of Linux, FreeBSD, and NetBSD support both O_SYNC and O_DIRECT making oflag=sync,direct portable among that set. | 23:49 |
| rwp | NetBSD does not support dd's "=fsync" for either conv or oflag making the fsync option not portable that direction. | 23:50 |
Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!