Databricks photon vs catalyst Optimizer

Question

I was reading about the new databricks feature of delta engine which is powered by photon. From the understanding i have by reading i have a feeling that its similar to catalyst optimizer. Will the photon replace the catalyst.

How is the photon engine different to catalyst optimizer

The link is referred below

https://databricks.com/blog/2020/06/24/introducing-delta-engine.html

Answer 1

I think you are conflating two things:

• Catalyst optimizer is about coming up "Steps to take to execute the query". For example, the optimizer will decide how and when to do the join, aggregations, filters etc. When it should do what. This is also called "Physical Plan" in technical parlance. (Catalyst optimizer is more for logical planning but that is a finer detail)
• The execution engine is actually the engine that carries out the steps decided by the optimizer. It does not second guess the steps, but rather makes sure to carry them out as efficiently as possible.

Delta Engine (or Photon) is an execution engine and not an optimizer. It is more directly comparable with the Tungsten (whole stage code generation) execution engine currently available in open source Spark. Both Tungsten and Delta-Engine are execution engines.