[Patch] Fish uploads files very slowly.

[buc <buc odusz so-cdu ru>]

  Midnight Commander has a FISH virtual filesystem (FIle transfer over
SHell). There is one old problem in MC implementation of it -- uploads
onto remote system.
  To do such an upload, MC should invoke on the remote side a command,
which reads appropriate number of bytes from stdin and store them in the
target file. Initially, `dd' command had been chosen for this purpose.



  The decision was such:
" ( dd bs=4096 count=<size/4096> ; dd bs=<size%4096> count=1 ) | ( cat
 >target_file ; cat >/dev/null ) "
  (Note: additional cat to /dev/null is needed to flush input on write
errors).
  But unfortunately, this variant appeared unreliable. The design of
`dd' does not assume full filling of input buffers on read (see, for
example, "conv=sync" description in `dd' manual). The design of `ssh' is
those that sometimes data can be written into pipe by portions of
different size. Therefore the part of data could remain not read by `dd' ...

  In the current version, the problem is solved as:
" dd bs=1 count=<size> | ..... "
i.e., `dd' reads input byte-by-byte. It is robast, but is very slow and
grabs a lot of cpu time on the remote system.



  The better decision is possible.
  There is `head' command. It has `-c <number>' option, which does what
we really need. The necessary amount of data is read from stdin reliably.
  Unfortunately, the remote system (any *NIX-like) may not have `head'
command, or have another implementation of it. But for such systems we
can still use `dd' .
                 The final decision is:
" ( head -c <size> -q - || dd bs=1 count=<size> ) | ( cat >target_file ;
cat >/dev/null ) "
  Either fast `head' is used, or `dd' as a fallback.



  If there is no `head' on the remote side, or `head' is too old (has no
`-c' option), then `dd' will be used. `-q' and `-' are used to cause
most non-GNU `head's to fail (by "incorrect" option). We can not trust
non-GNU implementations of `head' -- for example, AIX has `-c' option,
but always adds extra newline at the end of target file :-( . There is
small probability that remote `head' is not GNU head but supports all
our options. Let`s believe that it is "modern" head and it suits us! :-)



  The new variant of FISH uploads is successfully tested by us in
various environments (GNU-like, non-GNU) -- all works as it is expected.



  Appropriate patch attached.


--
		Dmitry K. Butskoj <dmitry butskoj name>
		Saint-Petersburg, Russia
		Red Hat Certified Engineer 809003662809495


diff -Nrbu mc-4.6.1-20041108/vfs/fish.c mc-4.6.1-20041108-OK/vfs/fish.c
--- mc-4.6.1-20041108/vfs/fish.c	2004-10-28 15:24:43.000000000 +0400
+++ mc-4.6.1-20041108-OK/vfs/fish.c	2004-11-24 17:10:26.000000000 +0300
@@ -502,7 +502,14 @@
 	close (h);
 	ERRNOR (EIO, -1);
     }
-    /* Use this as stor: ( dd block ; dd smallblock ) | ( cat > file; cat > /dev/null ) */
+    /* Use this as stor:
+     *  ( head -c number || dd bs=1 count=number ) | ( cat > file; cat >/dev/null ) 
+     *  If `head' is not present on the remote system, `dd' will be used.
+     * Unfortunately, we cannot trust most non-GNU `head' implementations
+     * even if `-c' options is supported. Therefore, we separate GNU head
+     * (and other modern heads?) using `-q' and `-' . This causes another
+     * implementations to fail (because of "incorrect options").
+     */
 
     print_vfs_message(_("fish: store %s: sending command..."), name );
     quoted_name = name_quote (name, 0);
@@ -514,24 +521,26 @@
 		 "> /%s\n"
 		 "echo '### 001'\n"
 		 "(\n"
-		   "dd bs=1 count=%lu\n"
+		   "head -c %lu -q - || dd bs=1 count=%lu\n"
 		 ") 2>/dev/null | (\n"
 		   "cat > /%s\n"
 		   "cat > /dev/null\n"
 		 "); echo '### 200'\n",
 		 (unsigned long) s.st_size, name, quoted_name,
+		 (unsigned long) s.st_size,
 		 (unsigned long) s.st_size, quoted_name);
     else
 	n = fish_command (me, super, WAIT_REPLY,
 		 "#STOR %lu /%s\n"
 		 "echo '### 001'\n"
 		 "(\n"
-		   "dd bs=1 count=%lu\n"
+		   "head -c %lu -q - || dd bs=1 count=%lu\n"
 		 ") 2>/dev/null | (\n"
 		   "cat >> /%s\n"
 		   "cat > /dev/null\n"
 		 "); echo '### 200'\n",
 		 (unsigned long) s.st_size, name,
+		 (unsigned long) s.st_size,
 		 (unsigned long) s.st_size, quoted_name);
 
     g_free (quoted_name);