On Fri, Mar 29, 2019 at 8:58 AM Jürg Billeter <
j bitron ch> wrote:
On Fri, 2019-03-29 at 16:45 +0900, Tristan Van Berkom wrote:
> On Fri, 2019-03-29 at 08:30 +0100, Jürg Billeter wrote:
> > Hi Benjamin,
> [...]
> > While we're changing the scheduler architecture, I'm wondering whether
> > we should also consider implementing/supporting a make job server (or
> > at least make sure it would be possible to support this in the future).
> > This would likely improve scheduling quite a bit for local execution.
> > And the push-based model would make this a much better fit, i.e., it
> > matches the direction we're going.
>
> Wouldn't it also make sense to consider job servers in the remote
> execution scenario ?
>
> I expect that most remote execution setups involve large servers where
> many remote workers are running.
Theoretically, yes. However, the BuildStream scheduler isn't running on
the remote workers, so this would be mostly independent. And it could
raise concerns of scheduling fairness across jobs from different
clients.
Exactly. BuildStream only has a job here as a remote execution _client_, it should submit _all_ the jobs that it can as soon as it can, and should then rely on the remote execution service to execute jobs efficiently and fairly.
> (That said, I think we're a bit off topic here and think we should be
> careful to not lump too many changes into the same activity, it just
> makes sense to consider how jobs servers fit into a future where this
> scheduler change happens).
Yes, I think it's worth considering when designing the new scheduler
architecture, however, actual job server support would most likely be a
separate (and optional) effort after the core changes.
+1.
Observation: there is some speculation based concern/fear in this thread. I prefer we assume that we are all trying to do the right thing in terms of architecture, code quality and actual usability (which includes performance). I am sure that we'll come across some difficult trade off calls we'll need to make along the way, but let's do that as we encounter them, rather than preemptively take a stance.
Cheers,
Jürg
Cheers,
Sander
_______________________________________________
buildstream-list mailing list
buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
