Distribute Task Execution (Nx Agents)

Applying Nx Affected and remote caching can significantly speed up your CI pipeline, but you might still hit limits. As your monorepo grows, the most effective way to maintain low CI times is by combining them with a smart strategy for parallelizing work across multiple machines. Manually setting up such distribution can be challenging, though, often resulting in suboptimal performance or requiring high maintenance over time.

Nx Cloud visualization of how tasks are being distributed with Nx Agents

Nx Agents offer several key advantages:

  • Declarative Configuration: No maintenance is required as your monorepo evolves, thanks to a declarative setup.
  • Efficient Task Replay: By leveraging remote caching, tasks can be replayed efficiently across machines, enhancing distribution speed.
  • Intelligent Task Distribution: Tasks are distributed based on historical run times and dependencies, ensuring correct and optimal execution.
  • Dynamic Resource Allocation: Agents are allocated dynamically based on the size of the PR, balancing cost and speed.
  • Seamless CI Integration: Easily adopt Nx Agents with your existing CI provider, requiring minimal setup changes.
  • Simple Activation: Enable distribution with just a single line of code in your CI configuration.

Enable Nx Agents

To enable task distribution with Nx Agents, make sure your Nx workspace is connected to Nx Cloud. If you haven't connected your workspace to Nx Cloud yet, run the following command:

npx nx connect

Check out the connect to Nx Cloud recipe for more details.

Then, adjust your CI pipeline configuration to enable task distribution. If you don't have a CI config yet, you can generate a new one using the following command:

npx nx g ci-workflow

The key line in your CI config is the start-ci-run command:

.github/workflows/ci.yml
1name: CI 2... 3 4jobs: 5 main: 6 runs-on: ubuntu-latest 7 steps: 8 ... 9 - uses: actions/checkout@v4 10 with: 11 fetch-depth: 0 12 13 - run: pnpm dlx nx-cloud start-ci-run --distribute-on="3 linux-medium-js" --stop-agents-after="build" 14 15 # Cache node_modules 16 - uses: actions/setup-node@v4 17 with: 18 node-version: 20 19 cache: 'pnpm' 20 ... 21 22 # Nx Affected runs only tasks affected by the changes in this PR/commit. Learn more: https://nx.dev/ci/features/affected 23 - run: pnpm exec nx affected -t lint test build 24

This command tells Nx Cloud to:

  • Start a CI run (npx nx-cloud start-ci-run)
  • Collect all Nx commands that are being issued (e.g., pnpm exec nx affected -t lint test build) and
  • Distribute them across 3 agents (3 linux-medium-js) where linux-medium-js is a predefined agent launch template.

Configure Nx Agents on your CI Provider

Every organization manages their CI/CD pipelines differently, so the guides don't cover org-specific aspects of CI/CD (e.g., deployment). They mainly focus on configuring Nx correctly using Nx Agents and Nx Replay.

How Nx Agents Work

Distribute Task Execution with Nx Agents

Nx Agents are declarative in that you only specify the number of agents and the type of agent you want to use. Nx Cloud then picks up the Nx commands that are being issued on your CI and distributes them automatically. This results in low maintenance and a much more efficient distribution strategy. A non-declarative approach would be one where you define which tasks or projects get executed on which machine, requiring you to adjust the configuration as your codebase changes.

Nx Agents uses a task-centric approach to distribution. Instead of defining which tasks run on which machine upfront, Nx Agents dynamically processes tasks based on availability and task dependencies/ordering. Imagine a stack of tasks that agents pick up based on their required processing time (from historical data) and task dependency/ordering (from the Nx graph). Not only is this more resource efficient, but it is also more resilient to failures since any other agent can pick up work if one agent fails during bootup. This method contrasts with traditional VM-centric approaches, where tasks must be predefined for specific machines, often leading to inefficiencies as your codebase grows. Read more on our blog post.

Nx Agents are cost and resource-efficient as the tasks are automatically distributed - optimizing for speed while also ensuring resource utilization is high and idle time is low. You can also dynamically adjust the number of agents based on the size of the PR, and we're working on some more AI-powered features to optimize this even further. In addition, remote caching guarantees tasks are not run twice, and artifacts are shared efficiently among agents.

Nx Agents are non-invasive in that you don't need to completely overhaul your existing CI configuration or your Nx workspace to use it. You can start using it with your existing CI provider by adding the nx-cloud start-ci-run... command mentioned previously. In addition, all artifacts and logs are played back to the main job so you can keep processing them as if they were run on the main job. Hence, your existing post-processing steps should still keep working as before.

For a more thorough explanation of how Nx Agents optimize your CI pipeline, read this guide to parallelization and distribution in CI.

Nx Agents Features

Relevant Repositories and Examples

By integrating Nx Agents into your CI pipeline, you can significantly reduce build times, optimize resource use, and maintain a scalable, efficient development workflow.