org.apache.spark.sql

functions

object functions

:: Experimental :: Functions available for DataFrame.

Annotations
@Experimental()
Since

1.3.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By inheritance
Inherited
  1. functions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. def abs(e: Column): Column

    Computes the absolute value.

    Computes the absolute value.

    Since

    1.3.0

  7. def acos(columnName: String): Column

    Computes the cosine inverse of the given column; the returned angle is in the range 0.

    Computes the cosine inverse of the given column; the returned angle is in the range 0.0 through pi.

    Since

    1.4.0

  8. def acos(e: Column): Column

    Computes the cosine inverse of the given value; the returned angle is in the range 0.

    Computes the cosine inverse of the given value; the returned angle is in the range 0.0 through pi.

    Since

    1.4.0

  9. def add_months(startDate: Column, numMonths: Int): Column

    Returns the date that is numMonths after startDate.

    Returns the date that is numMonths after startDate.

    Since

    1.5.0

  10. def approxCountDistinct(columnName: String, rsd: Double): Column

    Aggregate function: returns the approximate number of distinct items in a group.

    Aggregate function: returns the approximate number of distinct items in a group.

    Since

    1.3.0

  11. def approxCountDistinct(e: Column, rsd: Double): Column

    Aggregate function: returns the approximate number of distinct items in a group.

    Aggregate function: returns the approximate number of distinct items in a group.

    Since

    1.3.0

  12. def approxCountDistinct(columnName: String): Column

    Aggregate function: returns the approximate number of distinct items in a group.

    Aggregate function: returns the approximate number of distinct items in a group.

    Since

    1.3.0

  13. def approxCountDistinct(e: Column): Column

    Aggregate function: returns the approximate number of distinct items in a group.

    Aggregate function: returns the approximate number of distinct items in a group.

    Since

    1.3.0

  14. def array(colName: String, colNames: String*): Column

    Creates a new array column.

    Creates a new array column. The input columns must all have the same data type.

    Since

    1.4.0

  15. def array(cols: Column*): Column

    Creates a new array column.

    Creates a new array column. The input columns must all have the same data type.

    Annotations
    @varargs()
    Since

    1.4.0

  16. def array_contains(column: Column, value: Any): Column

    Returns true if the array contain the value

    Returns true if the array contain the value

    Since

    1.5.0

  17. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  18. def asc(columnName: String): Column

    Returns a sort expression based on ascending order of the column.

    Returns a sort expression based on ascending order of the column.

    // Sort by dept in ascending order, and then age in descending order.
df.sort(asc("dept"), desc("age"))
    Since

    1.3.0

  19. def ascii(e: Column): Column

    Computes the numeric value of the first character of the string column, and returns the result as a int column.

    Computes the numeric value of the first character of the string column, and returns the result as a int column.

    Since

    1.5.0

  20. def asin(columnName: String): Column

    Computes the sine inverse of the given column; the returned angle is in the range -pi/2 through pi/2.

    Computes the sine inverse of the given column; the returned angle is in the range -pi/2 through pi/2.

    Since

    1.4.0

  21. def asin(e: Column): Column

    Computes the sine inverse of the given value; the returned angle is in the range -pi/2 through pi/2.

    Computes the sine inverse of the given value; the returned angle is in the range -pi/2 through pi/2.

    Since

    1.4.0

  22. def atan(columnName: String): Column

    Computes the tangent inverse of the given column.

    Computes the tangent inverse of the given column.

    Since

    1.4.0

  23. def atan(e: Column): Column

    Computes the tangent inverse of the given value.

    Computes the tangent inverse of the given value.

    Since

    1.4.0

  24. def atan2(l: Double, rightName: String): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  25. def atan2(l: Double, r: Column): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  26. def atan2(leftName: String, r: Double): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  27. def atan2(l: Column, r: Double): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  28. def atan2(leftName: String, rightName: String): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  29. def atan2(leftName: String, r: Column): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  30. def atan2(l: Column, rightName: String): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  31. def atan2(l: Column, r: Column): Column

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Returns the angle theta from the conversion of rectangular coordinates (x, y) to polar coordinates (r, theta).

    Since

    1.4.0

  32. def avg(columnName: String): Column

    Aggregate function: returns the average of the values in a group.

    Aggregate function: returns the average of the values in a group.

    Since

    1.3.0

  33. def avg(e: Column): Column

    Aggregate function: returns the average of the values in a group.

    Aggregate function: returns the average of the values in a group.

    Since

    1.3.0

  34. def base64(e: Column): Column

    Computes the BASE64 encoding of a binary column and returns it as a string column.

    Computes the BASE64 encoding of a binary column and returns it as a string column. This is the reverse of unbase64.

    Since

    1.5.0

  35. def bin(columnName: String): Column

    An expression that returns the string representation of the binary value of the given long column.

    An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".

    Since

    1.5.0

  36. def bin(e: Column): Column

    An expression that returns the string representation of the binary value of the given long column.

    An expression that returns the string representation of the binary value of the given long column. For example, bin("12") returns "1100".

    Since

    1.5.0

  37. def bitwiseNOT(e: Column): Column

    Computes bitwise NOT.

    Computes bitwise NOT.

    Since

    1.4.0

  38. def broadcast(df: DataFrame): DataFrame

    Marks a DataFrame as small enough for use in broadcast joins.

    Marks a DataFrame as small enough for use in broadcast joins.

    The following example marks the right DataFrame for broadcast hash join using joinKey.

    // left and right are DataFrames
left.join(broadcast(right), "joinKey")
    Since

    1.5.0

  39. def callUDF(udfName: String, cols: Column*): Column

    Call an user-defined function.

    Call an user-defined function. Example:

    import org.apache.spark.sql._

val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id", "value")
val sqlContext = df.sqlContext
sqlContext.udf.register("simpleUDF", (v: Int) => v * v)
df.select($"id", callUDF("simpleUDF", $"value"))
    Annotations
    @varargs()
    Since

    1.5.0

  40. def cbrt(columnName: String): Column

    Computes the cube-root of the given column.

    Computes the cube-root of the given column.

    Since

    1.4.0

  41. def cbrt(e: Column): Column

    Computes the cube-root of the given value.

    Computes the cube-root of the given value.

    Since

    1.4.0

  42. def ceil(columnName: String): Column

    Computes the ceiling of the given column.

    Computes the ceiling of the given column.

    Since

    1.4.0

  43. def ceil(e: Column): Column

    Computes the ceiling of the given value.

    Computes the ceiling of the given value.

    Since

    1.4.0

  44. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  45. def coalesce(e: Column*): Column

    Returns the first column that is not null, or null if all inputs are null.

    Returns the first column that is not null, or null if all inputs are null.

    For example, coalesce(a, b, c) will return a if a is not null, or b if a is null and b is not null, or c if both a and b are null but c is not null.

    Annotations
    @varargs()
    Since

    1.3.0

  46. def col(colName: String): Column

    Returns a Column based on the given column name.

    Returns a Column based on the given column name.

    Since

    1.3.0

  47. def column(colName: String): Column

    Returns a Column based on the given column name.

    Returns a Column based on the given column name. Alias of col.

    Since

    1.3.0

  48. def concat(exprs: Column*): Column

    Concatenates multiple input string columns together into a single string column.

    Concatenates multiple input string columns together into a single string column.

    Annotations
    @varargs()
    Since

    1.5.0

  49. def concat_ws(sep: String, exprs: Column*): Column

    Concatenates multiple input string columns together into a single string column, using the given separator.

    Concatenates multiple input string columns together into a single string column, using the given separator.

    Annotations
    @varargs()
    Since

    1.5.0

  50. def conv(num: Column, fromBase: Int, toBase: Int): Column

    Convert a number in a string column from one base to another.

    Convert a number in a string column from one base to another.

    Since

    1.5.0

  51. def cos(columnName: String): Column

    Computes the cosine of the given column.

    Computes the cosine of the given column.

    Since

    1.4.0

  52. def cos(e: Column): Column

    Computes the cosine of the given value.

    Computes the cosine of the given value.

    Since

    1.4.0

  53. def cosh(columnName: String): Column

    Computes the hyperbolic cosine of the given column.

    Computes the hyperbolic cosine of the given column.

    Since

    1.4.0

  54. def cosh(e: Column): Column

    Computes the hyperbolic cosine of the given value.

    Computes the hyperbolic cosine of the given value.

    Since

    1.4.0

  55. def count(columnName: String): Column

    Aggregate function: returns the number of items in a group.

    Aggregate function: returns the number of items in a group.

    Since

    1.3.0

  56. def count(e: Column): Column

    Aggregate function: returns the number of items in a group.

    Aggregate function: returns the number of items in a group.

    Since

    1.3.0

  57. def countDistinct(columnName: String, columnNames: String*): Column

    Aggregate function: returns the number of distinct items in a group.

    Aggregate function: returns the number of distinct items in a group.

    Annotations
    @varargs()
    Since

    1.3.0

  58. def countDistinct(expr: Column, exprs: Column*): Column

    Aggregate function: returns the number of distinct items in a group.

    Aggregate function: returns the number of distinct items in a group.

    Annotations
    @varargs()
    Since

    1.3.0

  59. def crc32(e: Column): Column

    Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.

    Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.

    Since

    1.5.0

  60. def cumeDist(): Column

    Window function: returns the cumulative distribution of values within a window partition, i.

    Window function: returns the cumulative distribution of values within a window partition, i.e. the fraction of rows that are below the current row.

    N = total number of rows in the partition
cumeDist(x) = number of values before (and including) x / N

    This is equivalent to the CUME_DIST function in SQL.

    Since

    1.4.0

  61. def current_date(): Column

    Returns the current date as a date column.

    Returns the current date as a date column.

    Since

    1.5.0

  62. def current_timestamp(): Column

    Returns the current timestamp as a timestamp column.

    Returns the current timestamp as a timestamp column.

    Since

    1.5.0

  63. def date_add(start: Column, days: Int): Column

    Returns the date that is days days after start

    Returns the date that is days days after start

    Since

    1.5.0

  64. def date_format(dateExpr: Column, format: String): Column

    Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.

    Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.

    A pattern could be for instance dd.MM.yyyy and could return a string like '18.03.1993'. All pattern letters of java.text.SimpleDateFormat can be used.

    NOTE: Use when ever possible specialized functions like year. These benefit from a specialized implementation.

    Since

    1.5.0

  65. def date_sub(start: Column, days: Int): Column

    Returns the date that is days days before start

    Returns the date that is days days before start

    Since

    1.5.0

  66. def datediff(end: Column, start: Column): Column

    Returns the number of days from start to end.

    Returns the number of days from start to end.

    Since

    1.5.0

  67. def dayofmonth(e: Column): Column

    Extracts the day of the month as an integer from a given date/timestamp/string.

    Extracts the day of the month as an integer from a given date/timestamp/string.

    Since

    1.5.0

  68. def dayofyear(e: Column): Column

    Extracts the day of the year as an integer from a given date/timestamp/string.

    Extracts the day of the year as an integer from a given date/timestamp/string.

    Since

    1.5.0

  69. def decode(value: Column, charset: String): Column

    Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').

    Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.

    Since

    1.5.0

  70. def denseRank(): Column

    Window function: returns the rank of rows within a window partition, without any gaps.

    Window function: returns the rank of rows within a window partition, without any gaps.

    The difference between rank and denseRank is that denseRank leaves no gaps in ranking sequence when there are ties. That is, if you were ranking a competition using denseRank and had three people tie for second place, you would say that all three were in second place and that the next person came in third.

    This is equivalent to the DENSE_RANK function in SQL.

    Since

    1.4.0

  71. def desc(columnName: String): Column

    Returns a sort expression based on the descending order of the column.

    Returns a sort expression based on the descending order of the column.

    // Sort by dept in ascending order, and then age in descending order.
df.sort(asc("dept"), desc("age"))
    Since

    1.3.0

  72. def encode(value: Column, charset: String): Column

    Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').

    Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16'). If either argument is null, the result will also be null.

    Since

    1.5.0

  73. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  74. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  75. def exp(columnName: String): Column

    Computes the exponential of the given column.

    Computes the exponential of the given column.

    Since

    1.4.0

  76. def exp(e: Column): Column

    Computes the exponential of the given value.

    Computes the exponential of the given value.

    Since

    1.4.0

  77. def explode(e: Column): Column

    Creates a new row for each element in the given array or map column.

    Creates a new row for each element in the given array or map column.

    Since

    1.3.0

  78. def expm1(columnName: String): Column

    Computes the exponential of the given column.

    Computes the exponential of the given column.

    Since

    1.4.0

  79. def expm1(e: Column): Column

    Computes the exponential of the given value minus one.

    Computes the exponential of the given value minus one.

    Since

    1.4.0

  80. def expr(expr: String): Column

    Parses the expression string into the column that it represents, similar to DataFrame.

    Parses the expression string into the column that it represents, similar to DataFrame.selectExpr

    // get the number of words of each length
df.groupBy(expr("length(word)")).count()

  81. def factorial(e: Column): Column

    Computes the factorial of the given value.

    Computes the factorial of the given value.

    Since

    1.5.0

  82. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  83. def first(columnName: String): Column

    Aggregate function: returns the first value of a column in a group.

    Aggregate function: returns the first value of a column in a group.

    Since

    1.3.0

  84. def first(e: Column): Column

    Aggregate function: returns the first value in a group.

    Aggregate function: returns the first value in a group.

    Since

    1.3.0

  85. def floor(columnName: String): Column

    Computes the floor of the given column.

    Computes the floor of the given column.

    Since

    1.4.0

  86. def floor(e: Column): Column

    Computes the floor of the given value.

    Computes the floor of the given value.

    Since

    1.4.0

  87. def format_number(x: Column, d: Int): Column

    Formats numeric column x to a format like '#,###,###.

    Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places, and returns the result as a string column.

    If d is 0, the result has no decimal point or fractional part. If d < 0, the result will be null.

    Since

    1.5.0

  88. def format_string(format: String, arguments: Column*): Column

    Formats the arguments in printf-style and returns the result as a string column.

    Formats the arguments in printf-style and returns the result as a string column.

    Annotations
    @varargs()
    Since

    1.5.0

  89. def from_unixtime(ut: Column, f: String): Column

    Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.

    Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.

    Since

    1.5.0

  90. def from_unixtime(ut: Column): Column

    Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.

    Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.

    Since

    1.5.0

  91. def from_utc_timestamp(ts: Column, tz: String): Column

    Assumes given timestamp is UTC and converts to given timezone.

    Assumes given timestamp is UTC and converts to given timezone.

    Since

    1.5.0

  92. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  93. def greatest(columnName: String, columnNames: String*): Column

    Returns the greatest value of the list of column names, skipping null values.

    Returns the greatest value of the list of column names, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.

    Annotations
    @varargs()
    Since

    1.5.0

  94. def greatest(exprs: Column*): Column

    Returns the greatest value of the list of values, skipping null values.

    Returns the greatest value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.

    Annotations
    @varargs()
    Since

    1.5.0

  95. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  96. def hex(column: Column): Column

    Computes hex value of the given column.

    Computes hex value of the given column.

    Since

    1.5.0

  97. def hour(e: Column): Column

    Extracts the hours as an integer from a given date/timestamp/string.

    Extracts the hours as an integer from a given date/timestamp/string.

    Since

    1.5.0

  98. def hypot(l: Double, rightName: String): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  99. def hypot(l: Double, r: Column): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  100. def hypot(leftName: String, r: Double): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  101. def hypot(l: Column, r: Double): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  102. def hypot(leftName: String, rightName: String): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  103. def hypot(leftName: String, r: Column): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  104. def hypot(l: Column, rightName: String): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  105. def hypot(l: Column, r: Column): Column

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Computes sqrt(a2 + b2) without intermediate overflow or underflow.

    Since

    1.4.0

  106. def initcap(e: Column): Column

    Returns a new string column by converting the first letter of each word to uppercase.

    Returns a new string column by converting the first letter of each word to uppercase. Words are delimited by whitespace.

    For example, "hello world" will become "Hello World".

    Since

    1.5.0

  107. def inputFileName(): Column

    Creates a string column for the file name of the current Spark task.

  108. def instr(str: Column, substring: String): Column

    Locate the position of the first occurrence of substr column in the given string.

    Locate the position of the first occurrence of substr column in the given string. Returns null if either of the arguments are null.

    NOTE: The position is not zero based, but 1 based index, returns 0 if substr could not be found in str.

    Since

    1.5.0

  109. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  110. def isNaN(e: Column): Column

    Return true iff the column is NaN.

    Return true iff the column is NaN.

    Since

    1.5.0

  111. def lag(e: Column, offset: Int, defaultValue: Any): Column

    Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.

    Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row. For example, an offset of one will return the previous row at any given point in the window partition.

    This is equivalent to the LAG function in SQL.

    Since

    1.4.0

  112. def lag(columnName: String, offset: Int, defaultValue: Any): Column

    Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.

    Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row. For example, an offset of one will return the previous row at any given point in the window partition.

    This is equivalent to the LAG function in SQL.

    Since

    1.4.0

  113. def lag(columnName: String, offset: Int): Column

    Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row.

    Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row. For example, an offset of one will return the previous row at any given point in the window partition.

    This is equivalent to the LAG function in SQL.

    Since

    1.4.0

  114. def lag(e: Column, offset: Int): Column

    Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row.

    Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row. For example, an offset of one will return the previous row at any given point in the window partition.

    This is equivalent to the LAG function in SQL.

    Since

    1.4.0

  115. def last(columnName: String): Column

    Aggregate function: returns the last value of the column in a group.

    Aggregate function: returns the last value of the column in a group.

    Since

    1.3.0

  116. def last(e: Column): Column

    Aggregate function: returns the last value in a group.

    Aggregate function: returns the last value in a group.

    Since

    1.3.0

  117. def last_day(e: Column): Column

    Given a date column, returns the last day of the month which the given date belongs to.

    Given a date column, returns the last day of the month which the given date belongs to. For example, input "2015-07-27" returns "2015-07-31" since July 31 is the last day of the month in July 2015.

    Since

    1.5.0

  118. def lead(e: Column, offset: Int, defaultValue: Any): Column

    Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.

    Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row. For example, an offset of one will return the next row at any given point in the window partition.

    This is equivalent to the LEAD function in SQL.

    Since

    1.4.0

  119. def lead(columnName: String, offset: Int, defaultValue: Any): Column

    Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.

    Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row. For example, an offset of one will return the next row at any given point in the window partition.

    This is equivalent to the LEAD function in SQL.

    Since

    1.4.0

  120. def lead(e: Column, offset: Int): Column

    Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row.

    Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row. For example, an offset of one will return the next row at any given point in the window partition.

    This is equivalent to the LEAD function in SQL.

    Since

    1.4.0

  121. def lead(columnName: String, offset: Int): Column

    Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row.

    Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row. For example, an offset of one will return the next row at any given point in the window partition.

    This is equivalent to the LEAD function in SQL.

    Since

    1.4.0

  122. def least(columnName: String, columnNames: String*): Column

    Returns the least value of the list of column names, skipping null values.

    Returns the least value of the list of column names, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.

    Annotations
    @varargs()
    Since

    1.5.0

  123. def least(exprs: Column*): Column

    Returns the least value of the list of values, skipping null values.

    Returns the least value of the list of values, skipping null values. This function takes at least 2 parameters. It will return null iff all parameters are null.

    Annotations
    @varargs()
    Since

    1.5.0

  124. def length(e: Column): Column

    Computes the length of a given string or binary column.

    Computes the length of a given string or binary column.

    Since

    1.5.0

  125. def levenshtein(l: Column, r: Column): Column

    Computes the Levenshtein distance of the two given string columns.

    Computes the Levenshtein distance of the two given string columns.

    Since

    1.5.0

  126. def lit(literal: Any): Column

    Creates a Column of literal value.

    Creates a Column of literal value.

    The passed in object is returned directly if it is already a Column. If the object is a Scala Symbol, it is converted into a Column also. Otherwise, a new Column is created to represent the literal value.

    Since

    1.3.0

  127. def locate(substr: String, str: Column, pos: Int): Column

    Locate the position of the first occurrence of substr in a string column, after position pos.

    Locate the position of the first occurrence of substr in a string column, after position pos.

    NOTE: The position is not zero based, but 1 based index. returns 0 if substr could not be found in str.

    Since

    1.5.0

  128. def locate(substr: String, str: Column): Column

    Locate the position of the first occurrence of substr.

    Locate the position of the first occurrence of substr. NOTE: The position is not zero based, but 1 based index, returns 0 if substr could not be found in str.

    Since

    1.5.0

  129. def log(base: Double, columnName: String): Column

    Returns the first argument-base logarithm of the second argument.

    Returns the first argument-base logarithm of the second argument.

    Since

    1.4.0

  130. def log(base: Double, a: Column): Column

    Returns the first argument-base logarithm of the second argument.

    Returns the first argument-base logarithm of the second argument.

    Since

    1.4.0

  131. def log(columnName: String): Column

    Computes the natural logarithm of the given column.

    Computes the natural logarithm of the given column.

    Since

    1.4.0

  132. def log(e: Column): Column

    Computes the natural logarithm of the given value.

    Computes the natural logarithm of the given value.

    Since

    1.4.0

  133. def log10(columnName: String): Column

    Computes the logarithm of the given value in base 10.

    Computes the logarithm of the given value in base 10.

    Since

    1.4.0

  134. def log10(e: Column): Column

    Computes the logarithm of the given value in base 10.

    Computes the logarithm of the given value in base 10.

    Since

    1.4.0

  135. def log1p(columnName: String): Column

    Computes the natural logarithm of the given column plus one.

    Computes the natural logarithm of the given column plus one.

    Since

    1.4.0

  136. def log1p(e: Column): Column

    Computes the natural logarithm of the given value plus one.

    Computes the natural logarithm of the given value plus one.

    Since

    1.4.0

  137. def log2(columnName: String): Column

    Computes the logarithm of the given value in base 2.

    Computes the logarithm of the given value in base 2.

    Since

    1.5.0

  138. def log2(expr: Column): Column

    Computes the logarithm of the given column in base 2.

    Computes the logarithm of the given column in base 2.

    Since

    1.5.0

  139. def lower(e: Column): Column

    Converts a string column to lower case.

    Converts a string column to lower case.

    Since

    1.3.0

  140. def lpad(str: Column, len: Int, pad: String): Column

    Left-pad the string column with

    Left-pad the string column with

    Since

    1.5.0

  141. def ltrim(e: Column): Column

    Trim the spaces from left end for the specified string value.

    Trim the spaces from left end for the specified string value.

    Since

    1.5.0

  142. def max(columnName: String): Column

    Aggregate function: returns the maximum value of the column in a group.

    Aggregate function: returns the maximum value of the column in a group.

    Since

    1.3.0

  143. def max(e: Column): Column

    Aggregate function: returns the maximum value of the expression in a group.

    Aggregate function: returns the maximum value of the expression in a group.

    Since

    1.3.0

  144. def md5(e: Column): Column

    Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.

    Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.

    Since

    1.5.0

  145. def mean(columnName: String): Column

    Aggregate function: returns the average of the values in a group.

    Aggregate function: returns the average of the values in a group. Alias for avg.

    Since

    1.4.0

  146. def mean(e: Column): Column

    Aggregate function: returns the average of the values in a group.

    Aggregate function: returns the average of the values in a group. Alias for avg.

    Since

    1.4.0

  147. def min(columnName: String): Column

    Aggregate function: returns the minimum value of the column in a group.

    Aggregate function: returns the minimum value of the column in a group.

    Since

    1.3.0

  148. def min(e: Column): Column

    Aggregate function: returns the minimum value of the expression in a group.

    Aggregate function: returns the minimum value of the expression in a group.

    Since

    1.3.0

  149. def minute(e: Column): Column

    Extracts the minutes as an integer from a given date/timestamp/string.

    Extracts the minutes as an integer from a given date/timestamp/string.

    Since

    1.5.0

  150. def monotonicallyIncreasingId(): Column

    A column expression that generates monotonically increasing 64-bit integers.

    A column expression that generates monotonically increasing 64-bit integers.

    The generated ID is guaranteed to be monotonically increasing and unique, but not consecutive. The current implementation puts the partition ID in the upper 31 bits, and the record number within each partition in the lower 33 bits. The assumption is that the data frame has less than 1 billion partitions, and each partition has less than 8 billion records.

    As an example, consider a DataFrame with two partitions, each with 3 records. This expression would return the following IDs: 0, 1, 2, 8589934592 (1L << 33), 8589934593, 8589934594.

    Since

    1.4.0

  151. def month(e: Column): Column

    Extracts the month as an integer from a given date/timestamp/string.

    Extracts the month as an integer from a given date/timestamp/string.

    Since

    1.5.0

  152. def months_between(date1: Column, date2: Column): Column

  153. def nanvl(col1: Column, col2: Column): Column

    Returns col1 if it is not NaN, or col2 if col1 is NaN.

    Returns col1 if it is not NaN, or col2 if col1 is NaN.

    Both inputs should be floating point columns (DoubleType or FloatType).

    Since

    1.5.0

  154. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  155. def negate(e: Column): Column

    Unary minus, i.

    Unary minus, i.e. negate the expression.

    // Select the amount column and negates all values.
// Scala:
df.select( -df("amount") )

// Java:
df.select( negate(df.col("amount")) );
    Since

    1.3.0

  156. def next_day(date: Column, dayOfWeek: String): Column

    Given a date column, returns the first date which is later than the value of the date column that is on the specified day of the week.

    Given a date column, returns the first date which is later than the value of the date column that is on the specified day of the week.

    For example, next_day('2015-07-27', "Sunday") returns 2015-08-02 because that is the first Sunday after 2015-07-27.

    Day of the week parameter is case insensitive, and accepts: "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun".

    Since

    1.5.0

  157. def not(e: Column): Column

    Inversion of boolean expression, i.

    Inversion of boolean expression, i.e. NOT.

    // Scala: select rows that are not active (isActive === false)
df.filter( !df("isActive") )

// Java:
df.filter( not(df.col("isActive")) );
    Since

    1.3.0

  158. final def notify(): Unit

    Definition Classes
    AnyRef

  159. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  160. def ntile(n: Int): Column

    Window function: returns the ntile group id (from 1 to n inclusive) in an ordered window partition.

    Window function: returns the ntile group id (from 1 to n inclusive) in an ordered window partition. Fow example, if n is 4, the first quarter of the rows will get value 1, the second quarter will get 2, the third quarter will get 3, and the last quarter will get 4.

    This is equivalent to the NTILE function in SQL.

    Since

    1.4.0

  161. def percentRank(): Column

    Window function: returns the relative rank (i.

    Window function: returns the relative rank (i.e. percentile) of rows within a window partition.

    This is computed by:

    (rank of row in its partition - 1) / (number of rows in the partition - 1)

    This is equivalent to the PERCENT_RANK function in SQL.

    Since

    1.4.0

  162. def pmod(dividend: Column, divisor: Column): Column

    Returns the positive value of dividend mod divisor.

    Returns the positive value of dividend mod divisor.

    Since

    1.5.0

  163. def pow(l: Double, rightName: String): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  164. def pow(l: Double, r: Column): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  165. def pow(leftName: String, r: Double): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  166. def pow(l: Column, r: Double): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  167. def pow(leftName: String, rightName: String): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  168. def pow(leftName: String, r: Column): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  169. def pow(l: Column, rightName: String): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  170. def pow(l: Column, r: Column): Column

    Returns the value of the first argument raised to the power of the second argument.

    Returns the value of the first argument raised to the power of the second argument.

    Since

    1.4.0

  171. def quarter(e: Column): Column

    Extracts the quarter as an integer from a given date/timestamp/string.

    Extracts the quarter as an integer from a given date/timestamp/string.

    Since

    1.5.0

  172. def rand(): Column

    Generate a random column with i.

    Generate a random column with i.i.d. samples from U[0.0, 1.0].

    Since

    1.4.0

  173. def rand(seed: Long): Column

    Generate a random column with i.

    Generate a random column with i.i.d. samples from U[0.0, 1.0].

    Since

    1.4.0

  174. def randn(): Column

    Generate a column with i.

    Generate a column with i.i.d. samples from the standard normal distribution.

    Since

    1.4.0

  175. def randn(seed: Long): Column

    Generate a column with i.

    Generate a column with i.i.d. samples from the standard normal distribution.

    Since

    1.4.0

  176. def rank(): Column

    Window function: returns the rank of rows within a window partition.

    Window function: returns the rank of rows within a window partition.

    The difference between rank and denseRank is that denseRank leaves no gaps in ranking sequence when there are ties. That is, if you were ranking a competition using denseRank and had three people tie for second place, you would say that all three were in second place and that the next person came in third.

    This is equivalent to the RANK function in SQL.

    Since

    1.4.0

  177. def regexp_extract(e: Column, exp: String, groupIdx: Int): Column

    Extract a specific(idx) group identified by a java regex, from the specified string column.

    Extract a specific(idx) group identified by a java regex, from the specified string column.

    Since

    1.5.0

  178. def regexp_replace(e: Column, pattern: String, replacement: String): Column

    Replace all substrings of the specified string value that match regexp with rep.

    Replace all substrings of the specified string value that match regexp with rep.

    Since

    1.5.0

  179. def repeat(str: Column, n: Int): Column

    Repeats a string column n times, and returns it as a new string column.

    Repeats a string column n times, and returns it as a new string column.

    Since

    1.5.0

  180. def reverse(str: Column): Column

    Reverses the string column and returns it as a new string column.

    Reverses the string column and returns it as a new string column.

    Since

    1.5.0

  181. def rint(columnName: String): Column

    Returns the double value that is closest in value to the argument and is equal to a mathematical integer.

    Returns the double value that is closest in value to the argument and is equal to a mathematical integer.

    Since

    1.4.0

  182. def rint(e: Column): Column

    Returns the double value that is closest in value to the argument and is equal to a mathematical integer.

    Returns the double value that is closest in value to the argument and is equal to a mathematical integer.

    Since

    1.4.0

  183. def round(e: Column, scale: Int): Column

    Round the value of e to scale decimal places if scale >= 0 or at integral part when scale < 0.

    Round the value of e to scale decimal places if scale >= 0 or at integral part when scale < 0.

    Since

    1.5.0

  184. def round(e: Column): Column

    Returns the value of the column e rounded to 0 decimal places.

    Returns the value of the column e rounded to 0 decimal places.

    Since

    1.5.0

  185. def rowNumber(): Column

    Window function: returns a sequential number starting at 1 within a window partition.

    Window function: returns a sequential number starting at 1 within a window partition.

    This is equivalent to the ROW_NUMBER function in SQL.

    Since

    1.4.0

  186. def rpad(str: Column, len: Int, pad: String): Column

    Right-padded with pad to a length of len.

    Right-padded with pad to a length of len.

    Since

    1.5.0

  187. def rtrim(e: Column): Column

    Trim the spaces from right end for the specified string value.

    Trim the spaces from right end for the specified string value.

    Since

    1.5.0

  188. def second(e: Column): Column

    Extracts the seconds as an integer from a given date/timestamp/string.

    Extracts the seconds as an integer from a given date/timestamp/string.

    Since

    1.5.0

  189. def sha1(e: Column): Column

    Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.

    Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.

    Since

    1.5.0

  190. def sha2(e: Column, numBits: Int): Column

    Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.

    Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.

    e

    column to compute SHA-2 on.

    numBits

    one of 224, 256, 384, or 512.

    Since

    1.5.0

  191. def shiftLeft(e: Column, numBits: Int): Column

    Shift the the given value numBits left.

    Shift the the given value numBits left. If the given value is a long value, this function will return a long value else it will return an integer value.

    Since

    1.5.0

  192. def shiftRight(e: Column, numBits: Int): Column

    Shift the the given value numBits right.

    Shift the the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.

    Since

    1.5.0

  193. def shiftRightUnsigned(e: Column, numBits: Int): Column

    Unsigned shift the the given value numBits right.

    Unsigned shift the the given value numBits right. If the given value is a long value, it will return a long value else it will return an integer value.

    Since

    1.5.0

  194. def signum(columnName: String): Column

    Computes the signum of the given column.

    Computes the signum of the given column.

    Since

    1.4.0

  195. def signum(e: Column): Column

    Computes the signum of the given value.

    Computes the signum of the given value.

    Since

    1.4.0

  196. def sin(columnName: String): Column

    Computes the sine of the given column.

    Computes the sine of the given column.

    Since

    1.4.0

  197. def sin(e: Column): Column

    Computes the sine of the given value.

    Computes the sine of the given value.

    Since

    1.4.0

  198. def sinh(columnName: String): Column

    Computes the hyperbolic sine of the given column.

    Computes the hyperbolic sine of the given column.

    Since

    1.4.0

  199. def sinh(e: Column): Column

    Computes the hyperbolic sine of the given value.

    Computes the hyperbolic sine of the given value.

    Since

    1.4.0

  200. def size(e: Column): Column

    Returns length of array or map.

    Returns length of array or map.

    Since

    1.5.0

  201. def sort_array(e: Column, asc: Boolean): Column

    Sorts the input array for the given column in ascending / descending order, according to the natural ordering of the array elements.

    Sorts the input array for the given column in ascending / descending order, according to the natural ordering of the array elements.

    Since

    1.5.0

  202. def sort_array(e: Column): Column

    Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements.

    Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements.

    Since

    1.5.0

  203. def soundex(e: Column): Column

    * Return the soundex code for the specified expression.

    * Return the soundex code for the specified expression.

    Since

    1.5.0

  204. def sparkPartitionId(): Column

    Partition ID of the Spark task.

    Partition ID of the Spark task.

    Note that this is indeterministic because it depends on data partitioning and task scheduling.

    Since

    1.4.0

  205. def split(str: Column, pattern: String): Column

    Splits str around pattern (pattern is a regular expression).

    Splits str around pattern (pattern is a regular expression). NOTE: pattern is a string represent the regular expression.

    Since

    1.5.0

  206. def sqrt(colName: String): Column

    Computes the square root of the specified float value.

    Computes the square root of the specified float value.

    Since

    1.5.0

  207. def sqrt(e: Column): Column

    Computes the square root of the specified float value.

    Computes the square root of the specified float value.

    Since

    1.3.0

  208. def struct(colName: String, colNames: String*): Column

    Creates a new struct column that composes multiple input columns.

    Creates a new struct column that composes multiple input columns.

    Since

    1.4.0

  209. def struct(cols: Column*): Column

    Creates a new struct column.

    Creates a new struct column. If the input column is a column in a DataFrame, or a derived column expression that is named (i.e. aliased), its name would be remained as the StructField's name, otherwise, the newly generated StructField's name would be auto generated as col${index + 1}, i.e. col1, col2, col3, ...

    Annotations
    @varargs()
    Since

    1.4.0

  210. def substring(str: Column, pos: Int, len: Int): Column

    Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type

    Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type

    Since

    1.5.0

  211. def substring_index(str: Column, delim: String, count: Int): Column

    Returns the substring from string str before count occurrences of the delimiter delim.

    Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything the left of the final delimiter (counting from left) is returned. If count is negative, every to the right of the final delimiter (counting from the right) is returned. substring_index performs a case-sensitive match when searching for delim.

  212. def sum(columnName: String): Column

    Aggregate function: returns the sum of all values in the given column.

    Aggregate function: returns the sum of all values in the given column.

    Since

    1.3.0

  213. def sum(e: Column): Column

    Aggregate function: returns the sum of all values in the expression.

    Aggregate function: returns the sum of all values in the expression.

    Since

    1.3.0

  214. def sumDistinct(columnName: String): Column

    Aggregate function: returns the sum of distinct values in the expression.

    Aggregate function: returns the sum of distinct values in the expression.

    Since

    1.3.0

  215. def sumDistinct(e: Column): Column

    Aggregate function: returns the sum of distinct values in the expression.

    Aggregate function: returns the sum of distinct values in the expression.

    Since

    1.3.0

  216. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  217. def tan(columnName: String): Column

    Computes the tangent of the given column.

    Computes the tangent of the given column.

    Since

    1.4.0

  218. def tan(e: Column): Column

    Computes the tangent of the given value.

    Computes the tangent of the given value.

    Since

    1.4.0

  219. def tanh(columnName: String): Column

    Computes the hyperbolic tangent of the given column.

    Computes the hyperbolic tangent of the given column.

    Since

    1.4.0

  220. def tanh(e: Column): Column

    Computes the hyperbolic tangent of the given value.

    Computes the hyperbolic tangent of the given value.

    Since

    1.4.0

  221. def toDegrees(columnName: String): Column

    Converts an angle measured in radians to an approximately equivalent angle measured in degrees.

    Converts an angle measured in radians to an approximately equivalent angle measured in degrees.

    Since

    1.4.0

  222. def toDegrees(e: Column): Column

    Converts an angle measured in radians to an approximately equivalent angle measured in degrees.

    Converts an angle measured in radians to an approximately equivalent angle measured in degrees.

    Since

    1.4.0

  223. def toRadians(columnName: String): Column

    Converts an angle measured in degrees to an approximately equivalent angle measured in radians.

    Converts an angle measured in degrees to an approximately equivalent angle measured in radians.

    Since

    1.4.0

  224. def toRadians(e: Column): Column

    Converts an angle measured in degrees to an approximately equivalent angle measured in radians.

    Converts an angle measured in degrees to an approximately equivalent angle measured in radians.

    Since

    1.4.0

  225. def toString(): String

    Definition Classes
    AnyRef → Any

  226. def to_date(e: Column): Column

    Converts the column into DateType.

    Converts the column into DateType.

    Since

    1.5.0

  227. def to_utc_timestamp(ts: Column, tz: String): Column

    Assumes given timestamp is in given timezone and converts to UTC.

    Assumes given timestamp is in given timezone and converts to UTC.

    Since

    1.5.0

  228. def translate(src: Column, matchingString: String, replaceString: String): Column

    Translate any character in the src by a character in replaceString.

    Translate any character in the src by a character in replaceString. The characters in replaceString is corresponding to the characters in matchingString. The translate will happen when any character in the string matching with the character in the matchingString.

    Since

    1.5.0

  229. def trim(e: Column): Column

    Trim the spaces from both ends for the specified string column.

    Trim the spaces from both ends for the specified string column.

    Since

    1.5.0

  230. def trunc(date: Column, format: String): Column

    Returns date truncated to the unit specified by the format.

    Returns date truncated to the unit specified by the format.

    Since

    1.5.0

  231. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10](f: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8], arg9: scala.reflect.api.JavaUniverse.TypeTag[A9], arg10: scala.reflect.api.JavaUniverse.TypeTag[A10]): UserDefinedFunction

    Defines a user-defined function of 10 arguments as user-defined function (UDF).

    Defines a user-defined function of 10 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  232. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8, A9](f: (A1, A2, A3, A4, A5, A6, A7, A8, A9) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8], arg9: scala.reflect.api.JavaUniverse.TypeTag[A9]): UserDefinedFunction

    Defines a user-defined function of 9 arguments as user-defined function (UDF).

    Defines a user-defined function of 9 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  233. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8](f: (A1, A2, A3, A4, A5, A6, A7, A8) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8]): UserDefinedFunction

    Defines a user-defined function of 8 arguments as user-defined function (UDF).

    Defines a user-defined function of 8 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  234. def udf[RT, A1, A2, A3, A4, A5, A6, A7](f: (A1, A2, A3, A4, A5, A6, A7) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7]): UserDefinedFunction

    Defines a user-defined function of 7 arguments as user-defined function (UDF).

    Defines a user-defined function of 7 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  235. def udf[RT, A1, A2, A3, A4, A5, A6](f: (A1, A2, A3, A4, A5, A6) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6]): UserDefinedFunction

    Defines a user-defined function of 6 arguments as user-defined function (UDF).

    Defines a user-defined function of 6 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  236. def udf[RT, A1, A2, A3, A4, A5](f: (A1, A2, A3, A4, A5) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5]): UserDefinedFunction

    Defines a user-defined function of 5 arguments as user-defined function (UDF).

    Defines a user-defined function of 5 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  237. def udf[RT, A1, A2, A3, A4](f: (A1, A2, A3, A4) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4]): UserDefinedFunction

    Defines a user-defined function of 4 arguments as user-defined function (UDF).

    Defines a user-defined function of 4 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  238. def udf[RT, A1, A2, A3](f: (A1, A2, A3) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3]): UserDefinedFunction

    Defines a user-defined function of 3 arguments as user-defined function (UDF).

    Defines a user-defined function of 3 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  239. def udf[RT, A1, A2](f: (A1, A2) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2]): UserDefinedFunction

    Defines a user-defined function of 2 arguments as user-defined function (UDF).

    Defines a user-defined function of 2 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  240. def udf[RT, A1](f: (A1) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1]): UserDefinedFunction

    Defines a user-defined function of 1 arguments as user-defined function (UDF).

    Defines a user-defined function of 1 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  241. def udf[RT](f: () ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT]): UserDefinedFunction

    Defines a user-defined function of 0 arguments as user-defined function (UDF).

    Defines a user-defined function of 0 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

    Since

    1.3.0

  242. def unbase64(e: Column): Column

    Decodes a BASE64 encoded string column and returns it as a binary column.

    Decodes a BASE64 encoded string column and returns it as a binary column. This is the reverse of base64.

    Since

    1.5.0

  243. def unhex(column: Column): Column

    Inverse of hex.

    Inverse of hex. Interprets each pair of characters as a hexadecimal number and converts to the byte representation of number.

    Since

    1.5.0

  244. def unix_timestamp(s: Column, p: String): Column

    Convert time string with given pattern (see [http://docs.

    Convert time string with given pattern (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]) to Unix time stamp (in seconds), return null if fail.

    Since

    1.5.0

  245. def unix_timestamp(s: Column): Column

    Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the default timezone and the default locale, return null if fail.

    Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the default timezone and the default locale, return null if fail.

    Since

    1.5.0

  246. def unix_timestamp(): Column

    Gets current Unix timestamp in seconds.

    Gets current Unix timestamp in seconds.

    Since

    1.5.0

  247. def upper(e: Column): Column

    Converts a string column to upper case.

    Converts a string column to upper case.

    Since

    1.3.0

  248. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  249. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  250. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  251. def weekofyear(e: Column): Column

    Extracts the week number as an integer from a given date/timestamp/string.

    Extracts the week number as an integer from a given date/timestamp/string.

    Since

    1.5.0

  252. def when(condition: Column, value: Any): Column

    Evaluates a list of conditions and returns one of multiple possible result expressions.

    Evaluates a list of conditions and returns one of multiple possible result expressions. If otherwise is not defined at the end, null is returned for unmatched conditions.

    // Example: encoding gender string column into integer.

// Scala:
people.select(when(people("gender") === "male", 0)
  .when(people("gender") === "female", 1)
  .otherwise(2))

// Java:
people.select(when(col("gender").equalTo("male"), 0)
  .when(col("gender").equalTo("female"), 1)
  .otherwise(2))
    Since

    1.4.0

  253. def year(e: Column): Column

    Extracts the year as an integer from a given date/timestamp/string.

    Extracts the year as an integer from a given date/timestamp/string.

    Since

    1.5.0

Deprecated Value Members

  1. def callUDF(f: Function10[_, _, _, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column, arg9: Column, arg10: Column): Column

    Call a Scala function of 10 arguments as user-defined function (UDF).

    Call a Scala function of 10 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  2. def callUDF(f: Function9[_, _, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column, arg9: Column): Column

    Call a Scala function of 9 arguments as user-defined function (UDF).

    Call a Scala function of 9 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  3. def callUDF(f: Function8[_, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column): Column

    Call a Scala function of 8 arguments as user-defined function (UDF).

    Call a Scala function of 8 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  4. def callUDF(f: Function7[_, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column): Column

    Call a Scala function of 7 arguments as user-defined function (UDF).

    Call a Scala function of 7 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  5. def callUDF(f: Function6[_, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column): Column

    Call a Scala function of 6 arguments as user-defined function (UDF).

    Call a Scala function of 6 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  6. def callUDF(f: Function5[_, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column): Column

    Call a Scala function of 5 arguments as user-defined function (UDF).

    Call a Scala function of 5 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  7. def callUDF(f: Function4[_, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column): Column

    Call a Scala function of 4 arguments as user-defined function (UDF).

    Call a Scala function of 4 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  8. def callUDF(f: Function3[_, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column): Column

    Call a Scala function of 3 arguments as user-defined function (UDF).

    Call a Scala function of 3 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  9. def callUDF(f: Function2[_, _, _], returnType: DataType, arg1: Column, arg2: Column): Column

    Call a Scala function of 2 arguments as user-defined function (UDF).

    Call a Scala function of 2 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  10. def callUDF(f: Function1[_, _], returnType: DataType, arg1: Column): Column

    Call a Scala function of 1 arguments as user-defined function (UDF).

    Call a Scala function of 1 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  11. def callUDF(f: Function0[_], returnType: DataType): Column

    Call a Scala function of 0 arguments as user-defined function (UDF).

    Call a Scala function of 0 arguments as user-defined function (UDF). This requires you to specify the return data type.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use udf

    Since

    1.3.0

  12. def callUdf(udfName: String, cols: Column*): Column

    Call an user-defined function.

    Call an user-defined function. Example:

    import org.apache.spark.sql._

val df = Seq(("id1", 1), ("id2", 4), ("id3", 5)).toDF("id", "value")
val sqlContext = df.sqlContext
sqlContext.udf.register("simpleUDF", (v: Int) => v * v)
df.select($"id", callUdf("simpleUDF", $"value"))
    Annotations
    @deprecated
    Deprecated

    (Since version 1.5.0) Use callUDF

    Since

    1.4.0

Inherited from AnyRef

Inherited from Any

Aggregate functions

Collection functions

Date time functions

Math functions

Misc functions

Non-aggregate functions

Sorting functions

String functions

UDF functions

Window functions

Ungrouped