BenchmarkDotNet InProcessEmitToolchain Complete Sample

Question

I'm looking at BenchmarkDotNet and benchmarking in general for the first time ever. I appear to be unable to run benchmarks using the normal BenchmarkRunner because of antivirus restrictions on our work laptops so I'm trying to use InProcessEmitToolchain, as documented here. However, in those samples and the ones listed here I see no entry point for the application that will actually trigger the benchmarks and I've gotten nowhere useful reading through the documentation.

Can anyone point me at a complete sample of how to use the InProcessEmitToolchain and/or jobs that explains how to use jobs in conjunction with an application entry point to run the tests?

Answer 1

I was facing the same problem with antivirus (Windows Defender) blocking BenchmarkDotNet. I was able to figure out how to change the toolchain setup, though I had to use InProcessNoEmitToolchain since InProcessEmitToolchain was also blocked.

The example below did not actually trigger the antivirus, but it shows how to define which toolchain to use:

[Program.cs]

using BenchmarkDotNet.Running;
using Benchmarks;

_ = BenchmarkRunner.Run<MaterializeTest>();

[MaterializeTest.cs]

using BenchmarkDotNet.Attributes;

namespace Benchmarks;

[Config(typeof(AntiVirusFriendlyConfig))]
[MemoryDiagnoser]
public class MaterializeTest
{
    IEnumerable<int> _sequence;

    [Params(10, 100, 1000, 10000)]
    public int _size;

    [GlobalSetup]
    public void Setup()
    {
        _sequence = Enumerable.Range(0, _size).Select(i => Random.Shared.Next());
    }

    [Benchmark]
    public IReadOnlyList<int> ToList() => _sequence.ToList();

    [Benchmark]
    public IReadOnlyList<int> ToArray() => _sequence.ToArray();
}

[AntiVirusFriendlyConfig.cs]

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Toolchains.InProcess.NoEmit;

namespace Benchmarks;

public class AntiVirusFriendlyConfig : ManualConfig
{
    public AntiVirusFriendlyConfig()
    {
        AddJob(Job.MediumRun
            .WithToolchain(InProcessNoEmitToolchain.Instance));
    }
}

[Benchmarks.csproj]

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.13.2" />
  </ItemGroup>
</Project>

Answer 2

You can tweak the configuration with the toolchains you like:

var config = DefaultConfig.Instance
    .AddJob(Job
         .MediumRun
         .WithLaunchCount(1)
         .WithToolchain(InProcessEmitToolchain.Instance));

Or alternatively

var config = DefaultConfig.Instance
    .AddJob(Job
         .MediumRun
         .WithLaunchCount(1)
         .WithToolchain(InProcessNoEmitToolchain.Instance));

And pass it to the Run method (building on the program from Andreas Warberg)

_ = BenchmarkRunner.Run<MaterializeTest>(config);