Ryan Widmaier questions

Ryan Widmaier answers

• query vs filter and usage of correct expression in the query or filter
• Long running EMR cluster vs new cluster for each occurrence
• sum of case when in pyspark
• pyspark bitwiseAND vs ampersand operator
• Spark: Removing rows which occur less than N times
• Spark: executor memory exceeds physical limit
• Keeping a unused column in Spark when using group by?
• Achieve concurrency when saving to a partitioned parquet file
• How to reference a dataframe when in an UDF on another dataframe?
• elasticsearch scrolling using python client