.NET Core Web API and MongoDB Driver using Generic JSON Objects

Question

I'm creating an ASP.NET Web API to perform CRUD operations in a MongoDB database. I've been able to create a simple application based on the following Microsoft tutorial: Create a web API with ASP.NET Core and MongoDB.

This tutorial, like others I found, all use defined data models (in the above tutorial it's the Book model). In my case i need to perform CRUD operations with generic JSON objects. For example, the JSON object might be any of the following examples:

Example #1:

{_id: 1, name: 'Jon Snow', address: 'Castle Black', hobbies: 'Killing White Walkers'}

Example #2:

{_id: 2, name: 'Daenerys Targaryen', family: 'House Targaryen', titles: ['Queen of Meereen', 'Khaleesi of the Great Grass Sea', 'Mother of Dragons', 'The Unburnt', 'Breaker of Chains', 'Queen of the Andals and the First Men', 'Protector of the Seven Kingdoms', 'Lady of Dragonstone']}

The reason why I'm using a NoSQL database (MongoDB) is mainly because of the undefined data structure, and the ability to preform CRUD operations with just JSON.

As a trial and error attempt, I replaced the 'Book' model with 'object' and 'dynamic' but I get all sorts of errors regarding cast types and unknown properties:

public class BookService
{
    private readonly IMongoCollection<object> _books;

    public BookService(IBookstoreDatabaseSettings settings)
    {
        var client = new MongoClient(settings.ConnectionString);
        var database = client.GetDatabase(settings.DatabaseName);

        _books = database.GetCollection<object>(settings.BooksCollectionName);
    }

    public List<object> Get() => _books.Find(book => true).ToList();

    //public object Get(string id) => _books.Find<object>(book => book.Id == id).FirstOrDefault();

    //public object Create(object book)
    //{
    //    _books.InsertOne(book);
    //    return book;
    //}

    //public void Update(string id, object bookIn) => _books.ReplaceOne(book => book.Id == id, bookIn);

    //public void Remove(object bookIn) => _books.DeleteOne(book => book.Id == bookIn.Id);

    //public void Remove(string id) => _books.DeleteOne(book => book.Id == id);
}

Errors:

'object' does not contain a definition for 'Id' and no accessible extension method 'Id' accepting a first argument of type 'object' could be found (are you missing a using directive or an assembly reference?)

InvalidCastException: Unable to cast object of type 'd__51' to type 'System.Collections.IDictionaryEnumerator'.

So, my question is, how can I use generic JSON data types with ASP.NET Core Web API and the MongoDB Driver?

UPDATE: Based on @pete-garafano suggestion, I've decided to proceed with a POCO model.

I found an article in MongoDB's Github page explaining how to use static and dynamic data with the ASP.NET Core Driver. So I made the following changes to the Book model:

public class Book
{
    [BsonId]
    [BsonRepresentation(BsonType.ObjectId)]
    public string Id { get; set; }

    public string Name { get; set; }

    public decimal Price { get; set; }

    public string Category { get; set; }

    public string Author { get; set; }

    [BsonExtraElements]
    public BsonDocument Metadata { get; set; } //new property
}

Now I'm facing other issues, if my data is formatted exactly as the model, i'm able to list the data and create new entries in the database. But, if i try to create a new entry with the bellow format, i get an error:

{
    "Name": "test 5",
    "Price": 19,
    "Category": "Computers",
    "Author": "Ricky",
    "Metadata": {"Key": "Value"} //not working with this new field
}

System.InvalidCastException: Unable to cast object of type 'MongoDB.Bson.BsonElement' to type 'MongoDB.Bson.BsonDocument'.

Also, if i change the data format of one entry in Mongo and then try to list all results, i get the same error:

System.InvalidCastException: Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonBoolean'.

Based on the Mongo documents, the BsonExtraElements should allow generic/dynamic data to be attached to the model. What am I doing wrong in the new approach?

UPDATE #2: Added detailed stack trace of the error

System.InvalidCastException: Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonBoolean'. at get_AsBoolean(Object ) at System.Text.Json.JsonPropertyInfoNotNullable`4.OnWrite(WriteStackFrame& current, Utf8JsonWriter writer) at System.Text.Json.JsonPropertyInfo.Write(WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.JsonSerializer.HandleObject(JsonPropertyInfo jsonPropertyInfo, JsonSerializerOptions options, Utf8JsonWriter writer, WriteStack& state) at System.Text.Json.JsonSerializer.WriteObject(JsonSerializerOptions options, Utf8JsonWriter writer, WriteStack& state) at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonSerializer.WriteAsyncCore(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options, CancellationToken cancellationToken) at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding) at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters() --- End of stack trace from previous location where exception was thrown --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker) at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

UPDATE #3: Added the Book service and controller code files, database Book collection and exception launched in get() result.

BookServices.cs:

public class BookService
{
    private readonly IMongoCollection<Book> _books;

    public BookService(IBookstoreDatabaseSettings settings)
    {
        var client = new MongoClient(settings.ConnectionString);
        var database = client.GetDatabase(settings.DatabaseName);

        _books = database.GetCollection<Book>(settings.BooksCollectionName);
    }

    public List<Book> Get() => _books.Find(book => true).ToList();


    public Book Get(string id) => _books.Find<Book>(book => book.Id == id).FirstOrDefault();

    public Book Create(Book book)
    {
        _books.InsertOne(book);
        return book;
    }

    public void Update(string id, Book bookIn) => _books.ReplaceOne(book => book.Id == id, bookIn);

    public void Remove(Book bookIn) => _books.DeleteOne(book => book.Id == bookIn.Id);

    public void Remove(string id) => _books.DeleteOne(book => book.Id == id);
}

BooksController.cs:

[Route("api/[controller]")]
[ApiController]
public class BooksController : ControllerBase
{
    private readonly BookService _bookService;

    public BooksController(BookService bookService)
    {
        _bookService = bookService;
    }

    [HttpGet]
    public ActionResult<List<Book>> Get() => _bookService.Get(); // error happens when executing Get()

    [HttpGet("{id:length(24)}", Name = "GetBook")]
    public ActionResult<Book> Get(string id)
    {
        var book = _bookService.Get(id);

        if (book == null)
        {
            return NotFound();
        }

        return book;
    }

    [HttpPost]
    public ActionResult<Book> Create([FromBody] Book book)
    {
        _bookService.Create(book);

        return CreatedAtRoute("GetBook", new { id = book.Id.ToString() }, book);
    }

    [HttpPut("{id:length(24)}")]
    public IActionResult Update(string id, Book bookIn)
    {
        var book = _bookService.Get(id);

        if (book == null)
        {
            return NotFound();
        }

        _bookService.Update(id, bookIn);

        return NoContent();
    }

    [HttpDelete("{id:length(24)}")]
    public IActionResult Delete(string id)
    {
        var book = _bookService.Get(id);

        if (book == null)
        {
            return NotFound();
        }

        _bookService.Remove(book.Id);

        return NoContent();
    }
}

BookstoreDb.Books:

//non-pretty
{ "_id" : ObjectId("5df2b193405b7e9c1efa286f"), "Name" : "Design Patterns", "Price" : 54.93, "Category" : "Computers", "Author" : "Ralph Johnson" }
{ "_id" : ObjectId("5df2b193405b7e9c1efa2870"), "Name" : "Clean Code", "Price" : 43.15, "Category" : "Computers", "Author" : "Robert C. Martin" }
{ "_id" : ObjectId("5df2b1c9fe91da06078d9fbb"), "Name" : "A New Test", "Price" : 43.15, "Category" : "Computers", "Author" : "Ricky", "Metadata" : { "Key" : "Value" } }

Detailed result from Mongo Driver:

[/0]:{Api.Models.Book} Author [string]:"Ralph Johnson" Category [string]:"Computers" Id [string]:"5df2b193405b7e9c1efa286f" Metadata [BsonDocument]:null Name [string]:"Design Patterns" Price [decimal]:54.93

[/1]:{Api.Models.Book} Author [string]:"Robert C. Martin" Category [string]:"Computers" Id [string]:"5df2b193405b7e9c1efa2870" Metadata [BsonDocument]:null Name [string]:"Clean Code" Price [decimal]:43.15

[/2]:{Api.Models.Book} Author [string]:"Ricky" Category [string]:"Computers" Id [string]:"5df2b1c9fe91da06078d9fbb" Metadata [BsonDocument]:{{ "Metadata" : { "Key" : "Value" } }} AllowDuplicateNames [bool]:false AsBoolean [bool]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBoolean' threw an exception of type 'System.InvalidCastException' AsBsonArray [BsonArray]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonArray' threw an exception of type 'System.InvalidCastException' AsBsonBinaryData [BsonBinaryData]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonBinaryData' threw an exception of type 'System.InvalidCastException' AsBsonDateTime [BsonDateTime]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonDateTime' threw an exception of type 'System.InvalidCastException' AsBsonDocument [BsonDocument]:{{ "Metadata" : { "Key" : "Value" } }} AsBsonJavaScript [BsonJavaScript]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonJavaScript' threw an exception of type 'System.InvalidCastException' AsBsonJavaScriptWithScope [BsonJavaScriptWithScope]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonJavaScriptWithScope' threw an exception of type 'System.InvalidCastException' AsBsonMaxKey [BsonMaxKey]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonMaxKey' threw an exception of type 'System.InvalidCastException' AsBsonMinKey [BsonMinKey]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonMinKey' threw an exception of type 'System.InvalidCastException' AsBsonNull [BsonNull]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonNull' threw an exception of type 'System.InvalidCastException' AsBsonRegularExpression [BsonRegularExpression]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonRegularExpression' threw an exception of type 'System.InvalidCastException' AsBsonSymbol [BsonSymbol]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonSymbol' threw an exception of type 'System.InvalidCastException' AsBsonTimestamp [BsonTimestamp]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonTimestamp' threw an exception of type 'System.InvalidCastException' AsBsonUndefined [BsonUndefined]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsBsonUndefined' threw an exception of type 'System.InvalidCastException' AsBsonValue [BsonValue]:{{ "Metadata" : { "Key" : "Value" } }} AsByteArray [byte[]]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsByteArray' threw an exception of type 'System.InvalidCastException' AsDateTime [DateTime]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsDateTime' threw an exception of type 'System.InvalidCastException' AsDecimal [decimal]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsDecimal' threw an exception of type 'System.InvalidCastException' AsDecimal128 [Decimal128]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsDecimal128' threw an exception of type 'System.InvalidCastException' AsDouble [double]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsDouble' threw an exception of type 'System.InvalidCastException' AsGuid [Guid]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsGuid' threw an exception of type 'System.InvalidCastException' AsInt32 [int]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsInt32' threw an exception of type 'System.InvalidCastException' AsInt64 [long]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsInt64' threw an exception of type 'System.InvalidCastException' AsLocalTime [DateTime]:'(new System.Collections.Generic.ICollectionDebugView(test).Items2).Metadata.AsLocalTime' threw an exception of type 'System.InvalidCastException' [More] Name [string]:"A New Test" Price [decimal]:43.15

Answer 1

You should use BsonDocument to work with untyped data in C# with MongoDB.

private readonly IMongoCollection<BsonDocument> _books;

It isn't ideal, as C# would prefer strongly typed field names. I would recommend trying to build POCO models for the data to simplify query/update operations. If you cannot do this, you won't be able to use syntax like

_books.DeleteOne(book => book.Id == id);

You will need to instead use a dictionary type accesor syntax, like:

_books.DeleteOne(book => book["_id"] == id);

Please note that the _id field is special in MongoDB in that it must be present in every document, and unique in the collection. In the example you linked to, they provide an entity model. The Id field in this model has 2 decorators

[BsonId]
[BsonRepresentation(BsonType.ObjectId)]

These tell the driver that the Id field should be used as the _id, and that field, while a string in C# should be treated as an ObjectId by MongoDB.

If you are using a completely untyped model, you will need to be aware of the difference between _id and id and be sure to either map the fields properly or create an index on id (the former probably being what you want).

I wrote a post a while back that may be helpful to you. It covers much of the same material as the Microsoft post, but may provide you some more insight.

While the example data you provided does vary somewhat, it would still be possible to create a POCO model that would allow you to use type information in the queries. I would suggest that you investigate the feasibility of this to simplify your development. As I explained above, this isn't a requirement, but it will definitely improve the query experience.

Update To Address Extra Questions

The BsonExtraElements attribute is meant as a place for the driver to deserialize fields not in your model. For example, if you rename the Metadata field to, lets say, Foo, and rerun it. The Metadata field from the database should now actually be contained in the Foo field.

System.InvalidCastException: Unable to cast object of type 'MongoDB.Bson.BsonDocument' to type 'MongoDB.Bson.BsonBoolean'.

This exception seems to indicate something is a BsonDocument in the database, but the driver is trying to assign it to a bool value. I am unable to reproduce the error on my end. I created a document in the database just as you provided above.

I then queried using LINQPad and a simple program.

Can you perhaps provide the rest of the stack trace? It may provide us more information on which field is causing the problem. You can also try removing BsonExtraElements from the Metadata in your POCO and creating a new field just for BsonExtraElements.

Update 3

Thank you for providing the full stack trace. This lead me to an "a ha!" moment. The error is not coming from the MongoDB Driver, per se. The error is actually coming from the JSON Serializer as it accesses all the fields on the BsonDocument type.

A BsonDocument is a kind of lazy type. It doesn't "know" what it contains until you try to access it. This is handled by providing a getter for a number of different fields, all named by the type that it could contain. You can see them here.

The JSON Serializer in ASP is dutifully iterating over each field (AsBoolean, AsBsonArray, AsBsonBinaryData, etc) attempting to retrieve the value to serialize to JSON. Unfortunately, most of these are going to fail as the value in Metadata cannot cast to most (or any) of them.

I think you're going to need to either tell the JSON Serializer to ignore the Metadata field, or write a JSON serializer for BsonDocument.