apache-spark-sql tutorials

Why does Scala compiler fail with "no ': _*' annotation allowed here" when Row does accept varargs?

Jul 05, 2022

scala apache-spark apache-spark-sql

unexpected type: <class 'pyspark.sql.types.DataTypeSingleton'> when casting to Int on a ApacheSpark Dataframe

Nov 11, 2022

python apache-spark dataframe pyspark apache-spark-sql

How to overwrite entire existing column in Spark dataframe with new column?

Sep 18, 2022

apache-spark dataframe pyspark apache-spark-sql apache-spark-mllib

How do I get the last item from a list using pyspark?

Sep 18, 2022

apache-spark pyspark apache-spark-sql

SparkException: Values to assemble cannot be null

Oct 12, 2020

apache-spark apache-spark-sql apache-spark-ml

Convert timestamp to date in Spark dataframe

Feb 14, 2022

python python-3.x apache-spark pyspark apache-spark-sql

How to specify schema for CSV file without using Scala case class?

Nov 17, 2022

scala apache-spark apache-spark-sql

How to speed up Spark SQL unit tests?

Sep 17, 2022

unit-testing testing apache-spark apache-spark-sql

Spark 1.6: java.lang.IllegalArgumentException: spark.sql.execution.id is already set

Jul 11, 2022

scala apache-spark apache-spark-sql spark-dataframe

How do you create merge_asof functionality in PySpark?

Sep 17, 2022

python pandas apache-spark pyspark apache-spark-sql

Spark - java IOException :Failed to create local dir in /tmp/blockmgr*

Jan 04, 2022

hadoop apache-spark apache-spark-sql

pyspark using one task for mapPartitions when converting rdd to dataframe

Sep 17, 2022

python apache-spark pyspark apache-spark-sql

If I cache a Spark Dataframe and then overwrite the reference, will the original data frame still be cached?

Sep 17, 2022

python apache-spark pyspark apache-spark-sql

How does Spark SQL decide the number of partitions it will use when loading data from a Hive table?

Oct 29, 2022

apache-spark-sql

Preserve index-string correspondence spark string indexer

Apr 04, 2016

python apache-spark apache-spark-sql pyspark apache-spark-ml

Extract information from a `org.apache.spark.sql.Row`

Nov 13, 2022

scala apache-spark apache-spark-sql

How to run independent transformations in parallel using PySpark?

Sep 17, 2022

python-2.7 apache-spark pyspark apache-spark-sql python-multiprocessing

How to limit functions.collect_set in Spark SQL?

Aug 20, 2022

apache-spark apache-spark-sql

Why spark application fail with "executor.CoarseGrainedExecutorBackend: Driver Disassociated"?

Oct 31, 2022

apache-spark apache-spark-sql

How to subtract a column of days from a column of dates in Pyspark?

Sep 17, 2022

python apache-spark pyspark apache-spark-sql user-defined-functions

New posts in apache-spark-sql