spark-streaming tutorials

How to save latest offset that Spark consumed to ZK or Kafka and can read back after restart

Sep 20, 2022

Spark Streaming get warn "replicated to only 0 peer(s) instead of 1 peers"

May 19, 2021

java apache-spark streaming twitter4j spark-streaming

In Apache Spark, how to convert a slow RDD/dataset into a stream?

Sep 19, 2022

scala apache-spark apache-spark-sql spark-streaming

Spark Streaming: foreachRDD update my mongo RDD

Dec 08, 2019

mongodb apache-spark spark-streaming

Spark Streaming mapWithState seems to rebuild complete state periodically

Sep 19, 2022

scala apache-spark spark-streaming

How to fix "java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord" in Spark Streaming Kafka Consumer?

Jan 07, 2022

apache-spark serialization apache-kafka spark-streaming

What is the correct way to start/stop spark streaming jobs in yarn?

Sep 29, 2022

hadoop apache-spark spark-streaming hadoop-yarn cloudera

spark ssc.textFileStream is not streamining any files from directory

Oct 19, 2018

filesystems apache-spark spark-streaming data-stream

How to specify which java version to use in spark-submit command?

Aug 30, 2022

java hadoop-yarn spark-streaming

SBT Test Error: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream

Jun 25, 2022

scala apache-spark sbt spark-streaming scalatest

Amazon s3a returns 400 Bad Request with Spark

Nov 16, 2022

amazon-web-services amazon-s3 apache-spark hdfs spark-streaming

Spark off heap memory leak on Yarn with Kafka direct stream

Sep 06, 2020

apache-spark spark-streaming hadoop-yarn apache-spark-1.4

How to create Spark RDD from an iterator?

Oct 28, 2022

apache-spark spark-streaming

Kafka topic partitions to Spark streaming

Nov 22, 2020

apache-spark apache-kafka spark-streaming

java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ while running TwitterPopularTags

Jan 10, 2019

scala maven apache-spark noclassdeffounderror spark-streaming

Spark Dataframe validating column names for parquet writes

Aug 24, 2022

apache-spark pyspark apache-spark-sql spark-streaming parquet

Spark streaming with Kafka - createDirectStream vs createStream

Jun 21, 2019

apache-spark apache-kafka spark-streaming

How to stop spark streaming when the data source has run out

Sep 16, 2022

python apache-spark apache-kafka pyspark spark-streaming

AbstractMethodError creating Kafka stream

Mar 15, 2022

scala apache-spark apache-kafka spark-streaming

New posts in spark-streaming