How to bulk create or update in Django

Question

I have to process an item report CSV file every 1 hour. The CSV contains 150k+ records for 1 account and there are multiple accounts in my system. I was working previously on rails and there was active record gem to handle this use case very efficiently. I am looking for an alternate to this gem in Django or any built in method that will be helpful to import such large data in bulk.

So far I have tried this code.

class ItemReportService:

    def call(self, file_url):
        with open(file_url, 'r') as file:
            reader = csv.DictReader(file)
            products = []
            for row in reader:
                product = self.process_product(row)
                products.append(product)

            self.update_products(products)

    def process_product(self, row):
        print(f'Processing sku: {row["SKU"]}')
        product = Product.objects.filter(
            sku=row['SKU']).first() or Product(sku=row['SKU'])
        product.listing_title = row['Product Name']
        product.listed_price = row['Price']
        product.buy_box_price = row['Buy Box Item Price'] + \
            row['Buy Box Shipping Price']
        product.status = row['Lifecycle Status']
        return product

    def update_products(self, products):
        Product.objects.bulk_update(
            products,
            [
                'listing_title',
                'listed_price',
                'buy_box_price',
                'Lifecycle Status'
            ]
        )

It is raising this exception because when there is a new product it doesn't have primary key assigned to it

ValueError: All bulk_update() objects must have a primary key set.

Answer 1

Django 4.1 has new parameters for bulk_create(update_conflicts=bool and update_fields=[])

If your model has a field UNIQUE usually Django would ignore it when creating new data. But if you set the update_conflicts parameter to True, the fields inside update_fields will be updated.