apache-spark-sql tutorials

PySpark - Compare DataFrames

Feb 15, 2022

Processing multiple files as independent RDD's in parallel

May 09, 2022

scala apache-spark apache-spark-sql

Joining PySpark DataFrames on nested field

Oct 28, 2022

apache-spark dataframe join pyspark apache-spark-sql

How to ensure partitioning induced by Spark DataFrame join?

Jun 25, 2022

apache-spark dataframe join pyspark apache-spark-sql

Spark write to postgres slow

Oct 20, 2022

apache-spark dataframe apache-spark-sql

Peak Execution Memory in Spark

May 18, 2022

apache-spark apache-spark-sql

Find median in spark SQL for multiple double datatype columns

Oct 15, 2022

apache-spark apache-spark-sql hive-udf

Apache spark case with multiple when clauses on different columns

Jun 02, 2022

apache-spark hadoop apache-spark-sql

How to load a csv directly into a Spark Dataset?

Oct 23, 2022

scala apache-spark apache-spark-sql

merge two dataset which are having different column names in Apache spark

Aug 23, 2022

java apache-spark apache-spark-sql spark-dataframe

Why does spark-shell fail with "The root scratch dir: /tmp/hive on HDFS should be writable."?

Jul 31, 2022

apache-spark windows-10 apache-spark-sql

Why does a query fail with "AnalysisException: Expected only partition pruning predicates"?

May 15, 2019

apache-spark apache-spark-sql

What type should it be , after using .toArray() for a Spark vector?

Sep 10, 2022

python numpy apache-spark pyspark apache-spark-sql

Self-join not working as expected with the DataFrame API

Sep 05, 2022

scala apache-spark dataframe apache-spark-sql

Apply a transformation to multiple columns pyspark dataframe

Jun 29, 2022

apache-spark pyspark apache-spark-sql pyspark-sql

Is it possible to ignore null values when using lead window function in Spark

Feb 14, 2022

scala apache-spark apache-spark-sql

Does the SparkSQL Dataframe function explode preserve order?

Feb 11, 2021

scala apache-spark apache-spark-sql

How to sort array of struct type in Spark DataFrame by particular column?

Mar 16, 2022

dataframe scala apache-spark apache-spark-sql

Partitioning of Data Frame in Pyspark using Custom Partitioner

Aug 24, 2022

pyspark apache-spark-sql

How to expire state of dropDuplicates in structured streaming to avoid OOM?

Nov 21, 2022

apache-spark duplicates apache-spark-sql out-of-memory spark-structured-streaming

New posts in apache-spark-sql