On Thu, 2017-03-30 at 11:47 -0400, Colin Walters wrote:
On Wed, Mar 29, 2017, at 07:25 PM, Philip Withnall wrote:That makes sense. Using the ref name as an identifier would be a possibility, but I don't think it would work with Avahi, since a peer would have to advertise every ref it has,Why? Are you concerned about polling each remote for each ref? If the remote has a summary file, then we're talking about one HTTP request per remote. If it doesn't, things indeed get a bit uglier, but I suspect there are things we could do to optimize that, like remembering which Avahi nodes had any content we wanted before.
I'm not concerned about the cost of downloading the summary file from each remote. I am concerned about the cost of advertising a lot of refs in DNS-SD records, since the records are multicast across the local network, which gets expensive on Wi-Fi. Large mDNS adverts get split into multiple packets, increasing the probability of a collision and retransmission. For that reason, I don't think we can use ref names as identifiers for repository content, since if we did, we would have to include all the ref names in the DNS-SD records for a host --- which would be one ref for each flatpak app the host has installed (for example). There should be no need to cache DNS-SD query results in OSTree --- avahi-daemon does that (and it also obeys the myriad mDNS and DNS-SD cache coherency rules). --- How rigorous is the structure for naming refs? Would it be possible to use the common prefix of the refs as an advert? That would avoid having to generate some kind of UUID for the repository, but should also mean the DNS-SD records are relatively small, assuming that each repository typically contains high-related refs which share a common prefix. In that case, the process for working out which peers to download from would be (ignoring internet and locally-mounted repositories for the minute): 1. Do an Avahi query for the well-known OSTree service type (for example, _ostree._tcp) 2. Receive a DNS-SD record from each network peer in response, which contains the common prefix of all the refs it hosts, plus the timestamp of its summary file 3. Download the summary file each peer which advertised the ref (or a prefix of it) which we're interested in and which had a summary timestamp more recent than ours; verify that the summary is signed correctly and does contain the ref we care about 4. Download the commit metadata and signature for the commits mentioned in the downloaded summary files for the ref we care about (note that different peers might point to different commit checksums; the summary file doesn't contain a timestamp for each ref, as far as I can tell, so they all need to be downloaded) 5. Check the commit metadata, pick the most up to date one, then start pulling from all the peers which have that commit (i.e. we can now treat a subset of the peers as mirrors) Step 4 could be optimised by including the commit timestamp as additional metadata for each ref in the summary file, so we can order the commit checksums before downloading them. I don't know if summary files routinely contain these timestamps already (it doesn't look like the EOS ones do).
We currently have some code to cache the summary file which only kicks in if they're signed, but we could tweak that, then we'd do If-Modified-Since as well.
That would work well, and would complement what I put in step 2 about having the summary timestamp in a DNS-SD record. The benefit of putting the timestamp in the DNS-SD record is that the record can be re- advertised (broadcast to the local network) when a peer has updated its repository, which acts as a signal to everyone else on the network to update.
Yup. Though sorting by priority doesn't help when you're downloading the commit object; it should be done once the commits have all been downloaded, to address the case where the most recent commit is available via a slow transport and a fast one.We can definitely teach the fetcher code to prioritize file:// and Avahi over not-Avahi http://.
Ah, that would indeed speed things up in the case where the same commit checksum's metadata could be downloaded from several places. Am I missing any other situations it would speed up?
This can be sped up in the Avahi case by including the refspec and commit timestamp in DNS-SD records, so the summary file doesn't have to be download from all peers --- just the ones we choose to pull objects from. Though this would run into the oversized DNS-SD record problem I mentioned before.How many peers are we talking about here? Surely the cost of a HTTP request for the summary file isn't going to be large. We don't need to query *all* of them either - one optimization to make here is to include an "expected update time" in the commit metadata. If the current time is *before* that, we might only query one or two peers once a day. If it's *after* that, we could get increasingly more aggressive in trying to find an update via Avahi?
I'm thinking of roughly 30 peers on a local network, in a kind of classroom situation. We should probably design for a larger network though, to make things a bit more robust. Sorry if I didn't make it clear before: I don't think the cost of downloading the summary file is too much. The cost of having large DNS-SD records is too much. I'm not sold on the idea of an expected update time. That sounds like the kind of policy decision which should be made at a higher level in the stack (like gnome-software). When libostree is asked to resolve a repository to pull from, I think it should make a best effort at doing that.
I guess one thing that would help a lot too is to advertise the *timestamp* of the summary file in the DNS-SD record.
Yeah, definitely. One more thing to note is that DNS-SD records are not secure (you need DNSSEC for that --- and the interaction between DNSSEC and DNS-SD/mDNS is something I need to read up on), so we cannot entirely trust anything in a DNS-SD record. Before finalising any design, I would want to think about whether an attacker could prevent upgrades from being distributed by spoofing various DNS-SD records (or other things). But let's get a bit further with brainstorming/design first, and I'll put together a threat model once I've read a bit about mDNS spoofing.
But again, I'd like to understand why you think the cost of querying each remote for a ref would be too high.
Hopefully I've explained adequately above. I think some of the confusion might have stemmed from me not quite remembering exactly how metadata is split between the summary and commit metadata files. Hopefully the step-by-step example above of what I have in mind is a) correct and b) clear. :-) Philip
Attachment:
signature.asc
Description: This is a digitally signed message part