GNOME.org
 

Re: Matrix as a replacement for Telepathy

Hi Rob,
Thanks for the epic braindump here. I think that some of your caveats here may be more relevant to telepathy's specific model than Matrix, but to be honest I haven't really thought through (or discussed with the rest of the Matrix team) how best to solve our problem. Just to clarify the problem we're looking at solving:
* We already have a one true location for a user's conversations, contacts, presence, transferred files, notification settings, search indexes etc: it's their Matrix homeserver, which currently sits in a datacenter somewhere.
* However, accessing that server from a multiple different client apps on the same device (mobile or desktop) could start to get a bit gnarly. It's not a problem we have today, as in general people pick a single client (e.g. Riot or nheko or Quaternion or whatever) and stick with it. However, if Matrix was being used as a general communications function for an OS, we can see lots of different apps wanting to dip in to quickly send/receive messages, calls, handle Matrix push notifs etc. 

* So, the immediate options that spring to my mind are:
1. Each app independently acts as its own client to the server; either speaking the Matrix client-server protocol directly or by using one of the various Matrix client SDKs for the language of choice. This has the advantage of being dead easy (it's how it works already today), but the disadvantages of potentially redundant/duplicate data being synced to each client app over the network; potentially redundant processing (which can be quite complicated - e.g. e2e crypto); and redundant caching of client state (e.g. local copies of logs, e2e keys, etc).
2. We run a homeserver on the local device. This is the holy grail of Matrix dev; evolving servers to run clientside as well as serverside and effectively act as a hybrid p2p/client-server model. We seriously considered leaping ahead to this when we started writing our new homeserver implementation (Dendrite) but instead chose to nail the client-server model first rather than get ahead of ourselves. As such, it's at least a year away from happening.
3. We run a matrix client as a daemon which handles the tasks of e2e crypto, maintaining local conversation cache, etc. However, this would need to expose *some* kind of API through to other apps on the device. In turn, it could make app writers lives much simpler as they don't need to worry about e2e or other 'hard' stuff. It could potentially even handle WebRTC stuff (assuming the client app handed the daemon a window handle of some kind where it should be rendering video). The superficial similarity with the Telepathy architecture made me think of D-Bus as a possible way of doing this, but I really think the similarity is fairly superficial. 

With this in mind, further comments lie beneath:

On 05/09/2017 17:00, Robert McQueen wrote:
tl;dr - don't write multi-protocol messengers or gateways unless you have a hugely compelling reason, I did it for 10 years and all I got 
was this lousy t-shirt ;)
I'm not proposing a multi-protocol messenger here; just a way of getting at Matrix efficiently from a variety of different client apps. (Other protocols could conceivably sit behind the same API with some kind of shim to make them look like Matrix, but that's not what we're focused on here at all).
In terms of gateways: Matrix does have the concept of bridges, which operate strictly serverside. They act as gateways from Matrix to IRC, Slack, Gitter, Telegram, XMPP, etc. Our hugely compelling reason is that it's a major reason people use Matrix: to defragment communities (bridging them back together into a single Matrix room), and to act as a single UI for accessing a long tail of chatrooms that otherwise folks can't be bothered to run dedicated clients for. One specific instance is IRC bridges, where Matrix effectively just ends up acting as a big decentralised IRC bouncer; sort-of like a decentralised FOSS alternative to IRCCloud. For instance, right now we have around 20K active connections from the matrix.org IRC bridge deployments into ~10 different networks - this feels like a relatively substantial amount of the world's IRC traffic. This feels like a hugely compelling reason to me ;) 



On 25/08/17 17:43, Matthew Hodgson wrote:

Personally, I would really like to have a local comms API
abstraction, similar to the one that telepathy provided, but able
to support modern protocols - i.e. some daemon which exposes a
simple API for multiple apps to leverage when they want to
send/receive messages/calls/etc.  As I mentioned earlier, we need
it for the Librem 5 phone if nothing else (just as Maemo drove the
development of telepathy, iirc).
Having, er, done this already, for desktops as well as cellphones, I would really seriously encourage you to... not adopt this 
architecture. Most of the original goals underlying the Telepathy
architecture turned out to be false. You can't actually/realistically
unload the UI of the phone and have it be sufficiently
responsive/reliable for things like emergency calls, and you can't
(sensibly) implement reliable local logging of messages after you've
split the state of whether messages have been received, stored and
shown into three processes, and you can't efficiently implement a
contact list that aggregates presence and other metadata for contacts
unless you have all of the data in process, or copy it round
repeatedly (I can only apologise for libfolks).
As per the intro above, we're not suggesting precisely this architecture, although yes: we're considering storing state in both server, client-daemon and client-app. However, given Matrix's entire protocol is dedicated to the problem of syncing state from A to B over a federation of servers, this may not be quite so painful. 



You lose a huge amount by not being able to have your protocol code
in the same process as the UI, unless you write an API which
basically _is_ your protocol, or an API which is the UI, but the
bigger the gap between the two, the more developers fall into that
pit and don't get any UI or protocol work done. :)
So the API we're contemplating here /is/ the protocol. And we already have UIs which speak it (albeit talking to the server via HTTP, rather than to a local daemon via some appropriate RPC mechanism). 



Essentially, adding the D-Bus API to _entirely_ indirect the
protocol was, in the end, what made Telepathy so slow, expensive and
frustrating to develop, and discouraged any new contributions. As
you've rightly pointed out elsewhere in the thread, there is a _lot_
of protocol functionality that appeared even in XMPP at the time, as
well as other protocols since, which Telepathy never gained - part of
the problem is that there are now a long list of components from
spec, logger, client lib, UI, backend, etc that must all be developed
and tested together to add one vertical feature (eg - message read
receipts).
Right. This feels like the abstraction was too... abstract, rather than being coupled to a specific underlying protocol (i.e. Matrix). Or is D-Bus itself unperformant? 


Doing it over again, I'd just make "GNOME Chat" or whatever based on
a UI process which could optionally unload/hide it's UI. Then you
have single place on the system to do the common stuff like logging,
contact aggregation, protocol configuration, managing presence, etc.
If you _need_ to export API to integrate to the destkop, you can just
implement and export that. But that's really not all that different
to Pidgin/libpurple any more, maybe with a nicer GNOME-y UI - or a Matrix-only G* client which implemented everything you needed and had 
a few touch points.
Having the daemon being a headless Matrix client is precisely the idea here. I guess it could even show a UI itself in order to keep us honest, but given we already have multiple different Matrix clients speaking the Matrix client-server protocol, it feels a bit of a shame to not let the user pick their frontend UI.
Perhaps a better way of thinking of this is that the local daemon could effectively be a Matrix proxy (handling E2E and caching conversation history for clients which don't support it themselves). And then any old Matrix client could connect to it locally (perhaps via HTTP; perhaps by D-Bus or whatever) to benefit.
The only thing you lose vs current Telepathy in this scenario is the ostensible security benefits of having your protocols able to be sandboxed off into neat little walled gardens, although Telepathy was in theory architected this way I'm not entirely sure who outside of Nokia/Maemo managed to take advantage of this - possibly SELinux on Fedora/RH? 

However: with modern protocols and encryption approaches (just look
at Telepathy's chequered history with OTR) this falls down _really_
quickly when you need to do things like logging that must be either
trusted or not to access/store encrypted messages, potentially a
centralised key store, S/ZRTP where the media streaming code needs
the key anyway, etc etc - so in a sense I think some of the IM
protocols are too "high touch" to be segmented away like that - and
then the security is coming at a high cost of functionality.
Yeah. I imagine that one could always run multiple copies of my hypothetical matrix daemon to handle different matrix accounts, and so sandbox them nicely. Although something that does worry me about this approach is that it puts all your E2E eggs in one basket, and could allow any malicious privileged app on your box to suddenly start roaming around your decrypted E2E history, which would be a Very Bad Thing.
But in your "Matrix megadaemon" scenario, where you've added a huge complexity/ABI/IPC cost to changing UI <- IPC -> daemon, on top of 
the existing cost of gateway <- protocol -> client, so you've still
got all of these downsides, but none of the upside.
I'm not so sure. If one literally used the current Matrix HTTP API as the IPC between the client and the daemon, then it's potentially zero cost to the client. If one just mechanically mapped the HTTP API to a more idiomatic IPC mechanism, it'd hopefully be a fairly small additional cost.
Agreed that it adds another layer which has to be kept up-to-date with the latest & greatest Matrix features, but given we always fallback gracefully this doesn't feel like a showstopper.
I don't consider the cost of bridging to other protocols to be a consideration here, given in Matrix we do this serverside and it's all entirely isolated from this question of efficiently supporting multiple client apps on the same device. 




Neither Matrix nor libpurple provides this API out of the box
currently.


When I started IM client development in whenever it was - 2000 maybe
- the hot new thing in Gaim (now Pidgin) was the "core/UI split"
where by adding _magic IPC_ to Pidgin, a daemon would appear with all
the gnarly protocol/core/etc logic and then a plethora of UIs would
arrive on top of it. My recollection is that even without the IPC
(which I think didn't get past a "send this message to someone"
remote), the effort of cleaning up the internal interfaces to make
this even remotely possible burnt out the lead developer at the time,
although this did incidentally make libpurple a possibility.

Before starting Telepathy we also looked again at this, with "ok, now
we have D-Bus, can we just complete that rather than starting our
own spec?" and the answer was pretty clearly no, because there is a
lot of UI which is dependent on code in the protocol backends. With
the benefit of hindsight, that should've told us something - but we
weren't listening because we were busy trying to solve the
(imaginary) benefits of unloadable/componentised UIs, centralised
logging, etc. :)
Understood. Again, it feels like the emphasis here was on doing multi-head clients: finding a single UI which abstracts the various protocol backends - and so discovering there's lots of UI which is dependent on code in the protocol backends. For us, bridging really is an entirely separate problem (which does indeed suffer from these impedance mismatches by its very nature), but is completely separate to the problem of the API the client is using. 




So some replacement for telepathy probably is required, and if it
spoke both Matrix & libpurple & some native protocols on the
backend then so much the better.  Meanwhile, apps themselves can
link against libpurple matrix-*-sdk today according to taste, if
they want to use communication behaviour without relying on
anything provided by the OS.


I'd do it this way round: Make a list of use-cases which can't be met
by having _a_ client which speaks the protocol(s) you need, and then
figure out the places elsewhere in the desktop that need to access
it, and write API to be the "bottom of that UI" on a case by case
basis, rather than coming at it from the top of the protocol stack.
Right. We could absolutely do this: We pick a client (say: Riot) which speaks the protocol we need (Matrix), and figure out the places that the rest of the desktop needs to access it elsewhere (place calls, receive calls, show push notifs, let random apps send/receive messages), and expose an API which exposes those use cases whilst still letting Riot do its thing. 



Contributors will appreciate being able to develop on one component
and not 12, in one language, with less strict ABI/API concerns,
rapid, etc prototyping. David Woodhouse speaks good sense with his
lync/sipe battle stories, and from the UI side, in the end that
actually really killed off the ability for people to contribute to
Empathy even when they wanted to. "Hey can I add OTR support? There
is libotr!" well... bla bla bla new spec patch this patch that bla
bla security keys bla etc etc ok bye.
The horse has already bolted on this one, in that Matrix already has a slew of different clients and SDKs. But it doesn't seem to be discouraging contribs.
I agree that David's concerns about lync/sipe backends are not really covered by my proposal (given he wants to do the bridging clientside rather than serverside).
However... at the end of the day, after thinking about this for 15 years, I am now just massively skeptical about the multiple-protocol endeavour because it's trying to square a circle which is not a real-world problem, and it's antagonistic to the specific goals of 
every single IM/comms network provider, so you can burn resources to
win each battle but in the end, you will lose the war. It's appealing
for free software people because we're "the outsider" and we need
"our client" to access the "other people". But if you have one system
you prefer to use, and you are familiar with and aiming for that
specific system, you can _ALWAYS_ write a better client by ejecting
the other protocols and showing the semantics of that system
correctly to the user.
Agreed that bridging is always a compromise. This is why we have it partitioned off as its own thing which people are welcome to use and extend as much as they like... but in the end, if everyone gave up on running Matrix bridges, it'd still leave a pretty nice decentralised comms network. It's also worth noting that E2E encryption is effectively fundamentally incompatible with bridging. Even though OMEMO, Wire, Signal and Matrix all use variations of the same Double Ratchet Algorithm for their E2E, it obviously has to be bit-identical (complete with all the key management semantics) to bridge. So: we hedge our bets. Loads of people happily use Matrix<->IRC as a means of accessing IRC today. But we're not betting the farm on it.
Protocols differ, and the optimal UI for that differs too. Pidgin/libpurple has captured/retained the mindshare for people who 
want to add a protocol or need to develop UI along with their
protocol because the API allows them to do this without forcing
everything through a semantic abstraction. Any other user of
libpurple who isn't Pidgin has a commensurately (or significantly)
worse experience (just see telepathy-haze).

I'd wager that your Purism SMS and phone UI will be better at SMS
and phone if it simply doesn't try and do any of the other stuff -
the buttons you need in the UI are pretty much just set out in the
GSM spec. Android uses other apps to do the other stuff. The
exception is LTE SIP where it's literally GSM-semantic calls over a
different transport. Skype has a significantly different call UI for
PSTN calls than it does for Skype calls, because for usability's
sake, it's easier to represent different semantics to the user with a
different UI.
I'm not sure this is an argument for needing multiple apps. Totally agreed that PSTN calling should have different UX (e.g. a big grid of buttons) to IM, but this can absolutely be solved in the same app. We did it already (pre-Matrix) when building apps like blah.com. (For context, we've also been active in this space since ~2000, albeit originally focusing specifically on SIP & RTP and then briefly XMPP before prancing off into Matrix).
I think Polari is another excellent experiment which also proves the point - the best experience for IRC is an IRC client, which is as 
good an experience as we've written for people who want to use the
full range of functionality that IRC has to offer - but due to
Telepathy underneath, it doesn't _have_ all of that IRC
functionality, so it plateaus at a certain level of usefulness for
IRC users.
This is the same experience we see on Matrix today. For typical IRC users, the plateau is fine. For chanops or ircops it isn't, so they go fire up their native IRC client of choice. Which is a feature, not a bug, imo. 


Matrix might provide a better UI experience for people who wished
they weren't using IRC (as I believe many people here do), but the
gateway is definitely inferior for non-Matrix users as it fails to
correctly follow IRC norms. Matrix lets you redact a message even if
IRC has no way to send that, everyone has a [m] from the IRC
The [m] is just the default suffix; folks are welcome to set their own nick: i suspect you don't notice those who have ;) (We need to improve the UI massively for configuring bridged behaviour, which is on the relatively near horizon). 


and when
Matrix users speak to each other, the tab completion is for a
Matrix-derived full name rather than the IRC nick, making it hard for
IRC natives to follow a conversation, etc.


Yup, correctly mapping nicks is also on the horizon.
Not to pick on Matrix (or IRC), I just don't believe any longer that there's a silver bullet here. With hindsight, on the assumption that 
we want to design/control a certain level of experience, and
multi-protocol is a need for interoprtability even at the cost of
limiting functionality, I think we'd be better off investing as a
community into a GNOME Chat that used libpurple but knocked some of
the more weird API corners into submission, and exposed specific
well-designed integration points to the rest of the desktop - quick
chat/reply box in Shell, presence/etc for Contacts, file transfer in
Nautilus, etc.
The second you wrap libpurple you're surely back into multi-headed impedance mismatch problems again. 



This is in a sense a stylistic choice, but the Matrix protocol itself
in the end _must_ be an indirection/abstraction and hence developer friction and impedance mismatch point, the same as Telepathy's D-Bus 
API did, so although Matrix is well aligned with the IM/etc systems
of today, and I'm sure has avoided some of the conceptual mistakes
that got Telepathy tied up in latter days, I'm sure there are some
other different/new mistakes that mean the indirection will become increasingly expensive, difficult and frustrating as the world moves 
on.
Yup, I'm sure that we'll hit walls in future (although hopefully have designed the protocol to evolve around them). One thing's for sure though, which is that the richness of API we provide today is way beyond libpurple's (as much as I personally like libpurple).
And, same as XMPP - I was always a strong believer in the federation aspect and having an open/free protocol, but for two things, a) 
nobody uses it any more for IM and the big players cut their
portals/gateways off due to SPIM - after you abstract away identity
you can't model trust properly at scale any more, and
We are painfully aware of that (with ex-Talk/Hangouts/Wave folk hanging around the team). Fixing the reputation problem (and not abstracting away identity) is high on our hit list for 2017 - https://fosdem.org/2017/schedule/event/matrix_future/ etc. 


b) I don't
really like centralised gateways. :) You're essentially just adding a
centralised infrastructure problem to the semantic/abstraction one -
once XMPP became popular enough, the public servers and gateways
became a significant abuse target and the reliability totally
evaporated, because in the end every operator cares more about
"their" users than backing any 3rd party. I also don't really presume
that just because anyone can run a server that this is credible /
desirable for general users.
I'm not sure I'd describe Matrix's gateways as centralised: whilst matrix.org does run a bunch of them, we constantly bump into other ones run by others on the open network. Agreed that they are hard to run reliably.
If you're not trying to promote one network in favour of others, you don't really need to take on the challenge gateways to relegate 
certain protocols to 2nd tier - GNOME has a big enough fight on our
hands with incumbents and externalities frustrating us in the desktop
space without compounding the risk and joining a fight about the
winning IM protocol as well.
I'm certainly not suggesting GNOME gets into a fight over the best IM protocol.
It feels like the choice here is either: "Write a G* chat client built on libpurple for multiheadedness, which exposes some minimal APIs which other apps can piggyback on", or "Have apps talk Matrix to either a homeserver or some kind of local daemon/proxy". The two are not mutually exclusive. We're going to do the latter whatever, in some flavour, either for Purism or because we need it anyway (e.g. to add E2E support to apps who don't want to figure out E2E themselves). If it's something that's useful for the wider GNOME or freedesktop community then so much the better.
Given the requirements of folks like David who need something talking libpurple/SIPE to lync on the local host, perhaps both angles are sensible. Is there a reason why Pidgin itself couldn't fit the bill as a G* chat client? 

So, TL;DR: we agree in some places; we disagree in others ;)

Thanks again for taking the time to write it all up.

Matthew

--
Matthew Hodgson
Matrix.org
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]