Thanks for the pointer! I still don't understand 100%. The link states that tasks are parallelized automatically. At the same time, Dagger AIUI is intended to run on top of existing build servers like Jenkins, Circle CI or Github Actions. But then I would assume that some sort of integration between Dagger and the build server needs to be in place so that tasks are parallelized on multiple worker machines (rather than all running in parallel on a single machine). If everything is running inside of a single CircleCI job, that job doesn't have enough cores to run all the e2e tests in parallel.

I guess my question is how this integration works in practice and what kind of complexity it generates.

Good question. Let me provide a more detailed answer in terms of concurrency and parallelism. To paraphrase Rob Pike's excellent explanation:

- Concurrency is the breaking down of a program into pieces that can be executed independently.

- Parallelism is the simultaneous execution of multiple things (possibly related, possibly not)

Dagger is designed to be highly concurrent with minimal development effort. Compared to an equivalent configuration in a traditional CI system, your Dagger pipelines will be more concurrent, and require less lines of code.

Because it is highly concurrent, Dagger can be parallelized with relatively little effort. But as you pointed out, you still need to configure parallelization by setting up multiple nodes, etc. Dagger uses buildkit as an excution engine, so parallelizing Dagger boils down to parallelizing buildkit. There is a lot of work in this area (one benefit of building on an existing, mature ecosystem). For example, here is an example of Kubernetes deployment: https://github.com/moby/buildkit/tree/master/examples/kubern...

Note that, because of its highly concurrent design and because of built-in caching, a single node may get you further than you might think. For example, in the e2e testing example: instead of running every test for every new commit, like many CI systems do, Dagger will automatically cache tests with unchanged inputs. So, for example, a change in the documentation will not trigger the API tests; a change in the iOS app source code will not trigger the Android tests; etc.

It's not uncommong for CI runs to become faster after switching to Dagger, without any additional parallelism. As it turns out, most CIs are very wastesful and leave plenty of low-hanging fruits to pick. Dagger helps you pick them :)