apache-spark-sql tutorials

Memory issue when importing parquet files in Spark

Jan 24, 2022

OneHotEncoder in Spark Dataframe in Pipeline

Aug 20, 2022

scala apache-spark apache-spark-sql apache-spark-mllib apache-spark-ml

How to avoid boxing bytes in array in custom datasource?

Dec 15, 2018

scala apache-spark apache-spark-sql

How to convert unix timestamp to the given timezone with Spark

Feb 16, 2022

scala apache-spark apache-spark-sql timezone unix-timestamp

Retain raw JSON as column in Spark DataFrame on read/load?

Aug 25, 2022

json apache-spark apache-spark-sql

Why do I get so many empty partitions when repartionning a Spark Dataframe?

Nov 18, 2022

apache-spark pyspark apache-spark-sql partitioning

NOT IN implementation of Presto v.s Spark SQL

Oct 26, 2022

null apache-spark-sql presto isnull

Spark SQL - Regex for matching only numbers

Nov 10, 2022

regex dataframe apache-spark pyspark apache-spark-sql

Spark window partition function taking forever to complete

Sep 14, 2022

scala performance dataframe apache-spark apache-spark-sql

How to compare multiple rows?

Nov 06, 2019

scala apache-spark spark-streaming apache-spark-sql

Using groupBy in Spark and getting back to a DataFrame

Nov 02, 2022

scala apache-spark apache-spark-sql

How to get date and time from string?

Dec 06, 2018

scala date apache-spark apache-spark-sql

pyspark expected zero arguments for construction of ClassDict (for pyspark.mllib.linalg.DenseVector)

Dec 09, 2021

apache-spark pyspark apache-spark-sql user-defined-functions apache-spark-mllib

create hive external table with schema in spark

Nov 14, 2021

apache-spark hive apache-spark-sql spark-avro

How to GROUPING SETS as operator/method on Dataset?

Sep 10, 2022

apache-spark dataframe apache-spark-sql

PySpark: Get first Non-null value of each column in dataframe

Nov 03, 2022

python apache-spark dataframe pyspark apache-spark-sql

How to fill none values with a concrete timestamp in DataFrame?

Apr 22, 2022

apache-spark pyspark apache-spark-sql

PySpark - Compare DataFrames

Feb 15, 2022

python dataframe apache-spark pyspark apache-spark-sql

Processing multiple files as independent RDD's in parallel

May 09, 2022

scala apache-spark apache-spark-sql

Joining PySpark DataFrames on nested field

Oct 28, 2022

apache-spark dataframe join pyspark apache-spark-sql

New posts in apache-spark-sql