VSCode Extension Databricks-Connect - Use SparkSession

Question

I am using the Databricks VSCode extension for development in an IDE. The basic functionalities are all working well. I connected to an Azure Databricks workspace with Unity Catalog enabled, selected an active cluster (DBR 13.2) and configured the sync destination. I am able to execute code. Now I want to use Databricks Connect "V2" to run my code locally.

I have the following code:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

However, when I run this, I get the following error:

RuntimeError: Only remote Spark sessions using Databricks Connect are supported. Could not find connection parameters to start a Spark remote session.

Am I missing something? I did my authentication once with the AZ CLI, once with a PAT. I also tried it on DBR 13.2 and 13.3, but all options failed.

Thanks!

Answer 1

For anyone else that get's this error and does not wish to execute code using databricks.connect you have to completely uninstall the extension and package. I also recommend pip purge cache. Then you can reinstall the package and it should stop trying to use databricks.connect.