Is there a way to limit String Length in a spark dataframe Type?

Question

Is there to a way set maximum length for a string type in a spark Dataframe. I am trying to read a column of string, get the max length and make that column of type String of maximum length max length.

Is there a way to do this ?

Answer 1

There is no "limited length" string type in Spark. You can achieve the behavior via a transformation.

If you want long strings to be truncated, you can do this with something like:

val colName = "my_col"
val c = col(colName)
df.select(
  when(length(c) > maxLen, substring(c, 1, maxLen)).otherwise(c).as(colName)
)

If you want long strings to generate a runtime error, that is a bit more complicated, especially if you want readable error messages. You have to create a UDF that throws an error, e.g.,

/** Exception thrown by stop() UDF */
case class StopExecutionException(message: String) extends RuntimeException(message)

/**
 * Stops execution with a user defined error message.
 * This is useful when you want to stop processing due to an exceptional condition,
 * for example, an illegal value was encountered in the data.
 *
 * @param message the message of the exception: allows for data-driven exception messages
 * @tparam A return type to avoid analysis errors
 * @return the function never returns
 * @throws StopExecutionException
 */
def stop[A](message: String): A = {
  throw StopExecutionException(message)
}

val colName = ...
val c = col(colName)
df.select(
  when(length(c) <= maxLen, c)
    .otherwise {
      val stopUdf = udf(stop[String] _)
      stopUdf(concat(lit(s"Column $colName exceeds max length $maxLength: "), c))
    }
    .as(colName)
)

Last but not least, if you want to communicate maxLength metadata to a database so that it chooses an optimal storage type for short string columns, you have to add metadata to the dataframe column, e.g.,

val metadata = new MetadataBuilder().putLong("maxlength", maxLen).build()
df.select(c.as(colName, metadata))

Hope this helps.