Re: Your mc efforts is a great job !!! [Warning]: Long

[Pavel Roskin <proski gnu org>]

On Wed, 29 Oct 2003, Nikolai Bezroukov wrote:

> [Warning] Long, as I reiterated some points from the prev discussion.

Sorry, I could not reply earlier.  Valgrind shows that mc has serious
problems with garbage collection in VFS if extfs in involved.  I must give
priority to coding because it's something that nobody else will fix.

This discussion is something that others are more likely to participate
in.

> Pavel Roskin wrote:
>
> [snip]
>  > I'm not sure you are talking about mcfs. mcfs doesn't allow you to
>  > execute any remote commands other than whatever is already
>  > supported by mcserver. That's the standard file operations supported
> by FTP
>  > plus a few goodies like chmod.
>
> I do not know the details of the actual implementation, but IMHO the
> idea itself is very elegant although slightly ahead of the codebase with
> its hardwired hotkeys. It's better to be preserved and extended it to
> the full functionality. We should  not throw out the child with a water.

I think we are completely misunderstanding each other.  Hotkeys have
noting to do with it.  Either mcserv remains are a file server, or it
becomes a file server plus shell server (like sshd).  I believe we don't
need another shell server.  If you have a different opinion, please show
your code.  But if you don't document every function, I'm not going to
look at your patch.  There are enough good patches in the queue that I
don't have time to apply.

> Your concerns about security might be slightly overstated, as IPsec, SSL
> or similar secure or semi-secure channel can be used.  Moreover on
> switched networks that now prevail, the session is not that easy to
> sniff, if switches are configured correctly and hardened against ARP
> poisoning. For example in the 3COM switches (SuperStack II) enabling
> "Port Security" on a port makes the switch remember the first MAC
> address it receives and locks that MAC address to that port until
> overridden by manual intervention. After that any hacker can hit the
> wall with his head as long as he wish. Also any attempt to use ARP
> poisoning in corp. environment usually trigger IDS.

It's irresponsible to represent insecure servers as secure ones.  I know
that some distributions made mcserv a separate package, which is run by
default.  That was the main reason for disabling mcserv and mcfs by
default if installed.  We failed to explain distributors that mcserv is
insecure on public networks.  Disabling the code by default was the way to
get the message across.

I'm not going to compromise on security.  If you want insecure mcserv,
make it a separate project.  Port it to DOS, VxWorks, whatever.  I'm sure
embedded developers will like it.  GNU Midnight Commander will not enable
by default any inherently insecure code as long as I maintain it.

> Here I am discussing the general ("theoretical") value of client-server
> mode for this type of managers. And it is in this lights I stated in
> http://www.softpanorama.org/OFM/Ofm_04.shtml  and I am still convinced
> that mc client-server capabilities provides an important,  pioneering
> step in the right direction. If  keystrokes are parameterized into a
> "command language", not hardwired like in the current implementation,
> then this client server model is very natural with the server
> interpreting internal "command language" and client converting
> keystrokes to the command stream and updating panels.

mcfs is part of VFS.  This code is only enabled if VFS is enabled.  VFS is
"virtual filesystem".  It has nothing to do with keystrokes.  It presents
remote files, archives and other objects as files.  What you are
describing cannot be based on the existing mcfs code.

> The key advantage is the unique powerful and very attractive blend of
> FTP and telnet functionality in one interface. IMHO such a functionality
> might constitute a new protocol,  "a command line VNC", if you wish.

I'd rather have a blend of ssh and scp/sftp.  Currently we have problems
with ssh passwords.  They can be entered more by accident than by design.

> Probably equally important is the possibility of avoiding those horrible
> "Alice in Wonderland" key binding problems that litter this list. The
> server is immune to the host key bindings and you can use just one
> client on, say, the most "mc-friendly"  Linux flavor (RH8 with the
> gnome-terminal 2.0.1-5 ;-) to administer all your servers.  That
> instantly makes mc much more appealing tool to the system administrators
> who need to work with a lot of servers and might help to increase the
> development resources by attracting new talented developers.

I'm afraid you'll have to write that tool yourself.  And if it's insecure,
you will fix it or see it abandoned by "new talented developers".  Sane
people use ssh for that purpose.  By the way, you don't need to start
remote terminals, you can run ssh in a local terminal.

Support for file transfers over the existing connection could be
implemented in ssh itself if it's not implemented already.  I believe
there are serious security implications if the remove system is allowed to
upload or download files without specific approval from the user.  It
seems to me that making the local approval secure is the core of the
issue.  Nobody should be able to fake that prompt on any terminal.

>  > I was confused by "VFS" meaning two different things on adjacent
>  > lines. I still don't know what NC5 XTree VFS is. I have never used NC5.
>
> That's how I call "flat file VFS". As a PhD I have the responsibility to
> obscure things by inventing new terms ;-). See also
> http://www.softpanorama.org/OFM/Ofm_00.shtml.  The book discuses several
> implementations including NC5.

Actually, this should be implemented separately from VFS, as a higher
level.  It's representations of the tree on the panels, not
representations of something as a tree.

>  > > The idea is to have access to top and bottom directory in the
>  > > current panel (which of course depends of your sorting mode) via
>  > > hotkeys.
>  > As I said, I'm not aware of such key in mc.
>
> It's actually not defined either in OFM1999 standard or in OFM2004
> standard  (see http://www.softpanorama.org/OFM/Ofm_08.shtml and
> http://www.softpanorama.org/OFM/Ofm_09.shtml ). Here is what OFM1999 is
> saying  (sorry, it's slightly Windows-biased ;-) :
>
>     The hotkey assignment for sorting is optional.  If the hotkey is
>     defined its action should correspond to the F9-(LR)-S sequence for
>     the current panel and display the same menu so that the proper order
>     can be selected with the second key from the menu. Alternatively,
>     Ctrl-F3, Ctrl-F4, Ctrl-F5, Ctrl-F6 and Ctrl-F7 can be used for
>     sorting by name, extension, modification time, size and unsorted,
>     correspondingly ("NETSU" order).
>
> For Unix you need to add the ability to sort by sort by permissions,
> owner and group.  So the menu selection string became "NETSUPOG".
>
> As for the hotkey, actually, how about Ctrl-A ?  A very convenient
> hotkey for a frequently used operation  (and resorting of columns is
> definitely very frequently used, at least by advanced users). Actually
> you are in a better position to suggest a suitable hotkey as I do not
> really understand all the tradeoffs of the Unix environment especially
> the potential conflicts with Emacs/XEmacs (I use Vim/Gvim).

Please reread this.  I think you are misunderstanding me.  As I understand,
you wanted a hotkey to place the cursor on the first or last directory by
one keystroke.

I wrote you that we don't have such keys Actually, for the first
directory, Home is pretty close in most cases.  But suppose we have 1000
directories and 2000 files in some directory.  Then going to the last
_directory_ can be a boring task.

Now you are quoting text about sorting and asking me what I think about
using Ctrl-A for _sorting_, not for going to the last directory.

> I consider your efforts to be really outstanding because you not only
> managed to revitalize the development, but also stop the feature creep.
> Without your efforts mc probably would scale down to version 4.35.  IMHO
> two of your recent decisions  (1) to clean the code of GUI-related staff
> and (2) Windows-port related staff were very timely.

Thank you for your support.  I wish you were around when those features
were being added.

> Two things are now more or less clear:
>
>     -  mc is in the stabilization phase and it's level of maturity is
>     reasonably high,

This doesn't apply to VFS, although I hope to fix it.  Support for
background operations is a worse problem - it's useful, but I cannot be
implemented correctly without major changes.

I would say the code is in the "active stabilization" mode.  Sometimes the
only way to fix the code is to change it until it's clear what it does.
I'm not kidding, unfortunately.

>     -  the development resources are very scarce and just keeping up
>     with the usual flow of bugs strains the resources.

Yes.  Actually, there is some interest to "easy" parts of the code, but
it's often blocked by lower-level changes required for complete
implementation.

The code remains too complicated for many contributors to understand,
which leads to lower quality of patches.  Most functions are undocumented
or documented insufficiently.  What do you think can do a function called
vfs_add_noncurrent_stamps?  Function mc_def_ungetlocalcopy had just one
comment in version 4.6.0:

/* Dijkstra probably hates me... But he should teach me how to do this
nicely. */

Too bad, Dijkstra is not with us anymore.

> That's why I think it make sense to bite the bullet and move this
> "cleaning" strategy on a new level by using the capabilities of S-lang.
> Adding S-lang and exposing relevant internal functions might took the
> same amount of efforts as a regular cleaning of code that is needed
> anyway,  but with much better return on the investment. Iit gives you an
> opportunity to concentrate on simplifying and refactoring the core
> codebase by removing several ad hoc mini-languages invented in usual
> hacker-style traditions. For example, the code implementing "user
> extension predicates" and "user menu visibility predicates" are two
> examples of mini-languages, implementations of which can instantly be
> replaced with S-lang with no or very little effort.

I want to release a version that doesn't complain about "Invalid read of
size 4" in extfs_free().  Then I want to release a version that doesn't
have buffers of fixed sizes for paths.  I want to release a version that
has all contributed patches incorporated.

Please note that we don't currently include S-Lang interpreter, only
S-Lang screen library.  Also, it's possible to compile mc with ncurses
without S-Lang.  I believe ncurses support is worth keeping for now.

I'd like to see how you would handle something like the Open action for
HTML (search for "html" in mc.ext.in) - that action indeed has grown too
large.

> Also the pool of people who can develop/polish scriptable parts is
> bigger/deeper.  That's a natural separation of labor that the current
> development environment is lacking:  you need to be a jack of all trades
> and react to each and every hotkey problem (actually even in todays
> situation several things from the discussions can probably be added to
> FAQ to cut the number of such issues).  Assuming the core and scripting
> part are separated you have a core team and the extension development
> community like in Emacs. that's a more powerful combination and I
> suspect that more people might be willing to help with the scripting,
> but few would dig into the current C codebase.

I agree with your idea of separation of labor, but I don't see how it can
work with the current codebase.  Besides, all changes need to be reviewed,
and I don't see how switching to S-Lang would help.  I don't know any
S-Lang programmers.

Also I'm afraid you are overestimating the number of problems with
keys that can be fixed by making them scriptable.  Many problems are
inherent to terminals.

> Moreover S-lang usage in mc may provide stimulus for further improvement
> of both the S-lang and its interpreter and thus lift some or all of the
> current limitations/reservations. There also may be some kind of
> cross-pollination / integration with jed, which is a much better editor
> then cooledit , so this is actually not a one time tradeoff when you can
> lose of win, but a dynamic situation, a potentially mutually beneficial
> two-way process.

My experience shows that "cross-pollination" doesn't work.  We had
Cooledit code in mc.  There were patches sent to the mc lists with fixes
for the editor.  Some were specific to the text mode, some were not.
Nobody cared to forward those patches to the Cooledit maintainer.  It
takes time.  When people contribute code for free, it's hard to ask them
to do extra efforts, especially with software they don't use (i.e.
graphical Cooledit).

Neither did "cross-pollination" work for Samba.  Changes in vfs/samba are
not contributed to the Samba project.  In fact, the code we are including
has become a library of its own, based on old Samba code.

> It is true that S-lang has its own warts and limitations, but please
> understand that is such situation there is no best choice, only the
> worst choice and this might be inaction.  The truth is that even the
> worst choice (let's say adopting bash ;-) is better than none: scripting
> language is one of the few available avenues of addressing the feature
> creep problem .

I really don't care that much about the programming language for the
scripts as long as it works.

I suggested using bash for mc.ext, but others in this list didn't like the
idea.  I don't remember the reasons, but it should be archived.

The real question is whether you want to make mc.ext and mc.menu single
programs or not.  If they are single programs, then the choice of the
language doesn't matter - mc will be communicating with a program.

If you want mc.ext and mc.menu to be sets of scripts that are parsed by
mc, then I can say that they are already written in bash.

> The other solution might be exposing API and introducing a FAR-style
> plug-ins mechanism, although they have their own problems and having a
> two dozens or more of them installed lead to "overcrowding of the
> kitchen". But technically this is as simple as introducing plug-in menu
> in addition to the user menu or option to call a plug-in in the user menu.

This would lock the API at the time when significant reorganization of
code is underway.

> If the recent crisis, that you resolved by stepping in, is any
> indication about the future, a large pure-C open-source implementation
> like mc might not be sustainable due to the level of complexity and
> those of them, which cannot not attract enough resources need either to
> scale down of face the possibility of extinction. Just because the pool
> of really knowledgeable C developers is shrinking and they are in fact
> pretty close to becoming an endangered specie.  We all like free lunch
> but somebody needs to heat it even if the food is free ;-)

There are so many issues here, let me comment on them separately.

Pure C.  Yes, I appreciate your insight.  Even slightly "impure" C, like C
with gcc extensions, could be helpful.  Linux kernel is an example.  It's
well structured thanks to macros, initializers of structures, constructors
and explicit exports.

But in fact, the code is screaming for a language with OOP support.  A lot
of incorrect interdependencies would have been avoided in the compiler was
enforcing some rules.  It's especially important for collaborative
projects.

But on the other hand we have Gtk, which is pure C and sustainable.
Maybe it's because Gtk developers avoided feature bloat from the beginning
and cared about code quality.  But it's a library - you are guaranteed to
have many programmers using it.  GNU Midnight Commander is for end users,
even if they happen to be e.g. Perl programmers.  Besides, Gtk is used in
GUI.

I know programmers that only work in GUI.  There was the punchcard
generation, the command line generation, the text UI generation (TUI), and
now we have the GUI generation of programmers.  GNU Midnight Commander
serves only the text UI generation of users, and so it's not appealing to
the young users and programmers.  It's a niche project.

There are 3 large free office suites in development, so the programmers
resources are not scarce.  Those who started with Microsoft Word are young
and have enough time.  Those who started with Norton Commander have
serious jobs and in many cases families.  We are endangered species - the
text UI programmers with enough time for free software that one won't put
on resume like Linux kernel, Apache or cryptography.  But let's not
generalize.

> That's why making this decision and taking some steps now, when the mc
> development is clearly revitalized because of your efforts is, in my
> opinion, so important.

OK, I'm afraid I cannot spend any more time on the discussion unless you
show me something more specific.

-- 
Regards,
Pavel Roskin