Hangfire ContinueWithJob is stuck in awaiting state, though parent job has succeeded

Question

I have a few jobs executed one after the other via ContinueJobWith<MyHandler>(parentJobId, x => x.DoWork()).

However, the second job is not getting processed and always sits in Awaiting state:

The job itself is like this:

Why this can happen and where to check for resultion?

• We are using Autofac as DI container, but we have our own JobActivator implementation because we have to deal with multitenancy.
• We are using SQL Server 2019 for storage.
• Hangfire version is 1.7.10
• This is MVC 5 application
• I've not seen any errors/exceptions in any logs or during debugging
• After going through this I've added this to our Autofac registration

            builder.RegisterType<BackgroundJobStateChanger>()
                   .As<IBackgroundJobStateChanger>()
                   .InstancePerLifetimeScope();

This made no difference.

This is how the jobs are executed:

var parentJobId = _backgroundJobClient.Schedule<Handler>(h => h.ConvertCertToTraining(certId, command.SetUpOneToOneRelationship), TimeSpan.FromSeconds(1));
var filesCopyJObId = _backgroundJobClient.ContinueJobWith<Handler>(parentJobId, h => h.CopyAttachedFiles());
_backgroundJobClient.ContinueJobWith<Handler>(filesCopyJObId, h => h.NotifyUser(command.CertificationToBeConvertedIds, _principal.GetEmail()));

All the parameters are either int, bool or string. If I enqueue the awaiting jobs by hand, they are executed without issues.

I've added Hangfire logging, but could not see any issues there: server starts, stops, jobs change status, but could not see any obvious errors there.

What other things I should consider or where/how should I debug this?

Answer 1

From the looks of it, the first job with ID 216348 completed successfully but your second job with ID 216349 is waiting on the parent ID of 216347. According to Hangfire documentation and experience, the parentID should be of the job that you are waiting to finish before executing the second job.

According to Hangfire documentation on ContinueJobWith, "Continuations are executed when its parent job has been finished". From your screenshots, it is not clear whats going on with JobID: 216347. Once this job, 216347 completes, job with ID 216349 should kick off. If you are expecting 216349 to start after 216348 finishes, check your code and make sure correct ParentID is passed to the second job.

---

Update

Based on this thread, add the ContinuationsSupportAttribute to GlobalJobFilters.Filter where you configure Hangfire service. This should make your Hangfire instance aware of continuation jobs.

GlobalJobFilters.Filters.Add(new ContinuationsSupportAttribute());

Answer 2

During the investigation, it turned out that we were replacing JobFilterProviderCollection with our own collection:

    var filterProviderCollection = new JobFilterProviderCollection
    {
        new MyFilterProvider(...)
    };
    var backgroundJobClient = new BackgroundJobClient(JobStorage.Current, filterProviderCollection);

MyFilterProvider looked like this:

    public IEnumerable<JobFilter> GetFilters(Job job)
    {
        return new JobFilter[]
        {
             new JobFilter(new HangfireTenantFilter(_tenantDetail, _principal), JobFilterScope.Global,  null),
             new JobFilter(new HangfireFunctionalityFilter(_functionalityFilter), JobFilterScope.Global, null),
        };
    }

It turned out that code that was doing work on Continuation only took filters from this filter collection and ContinuationsSupportAttribute was not executed there in the right time. So re-adding default Hangfire filters from GlobalJobFilters.Filters fixed the situation:

    public IEnumerable<JobFilter> GetFilters(Job job)
    {
        var customFilters = new List<JobFilter>()
        {
             new JobFilter(new HangfireTenantFilter(_tenantDetail, _principal), JobFilterScope.Global,  null),
             new JobFilter(new HangfireFunctionalityFilter(_functionalityFilter), JobFilterScope.Global, null),
        };
        customFilters.AddRange(GlobalJobFilters.Filters);

        return customFilters;
    }