Skip to content
Advertisement

Facing issue while writing SQL in pyspark

I am trying to convert below SQL code to pyspark. Can someone please help me

UPDATE B 
SET col = 'a'
FROM table B
WHERE (case when left(util,1) in ('A', 'B') then count  else PROCS end < 1)

Here, util, count, procs are column names.

while coding in pyspark, i can create a new column ‘col’ like this:

df1 = df1.withColumn("col", case condition ,a)

Advertisement

Answer

You can use when for doing the equivalent of update:

df1 = df1.withColumn(
    'col', 
    F.when(
        F.expr("case when left(util,1) in ('A', 'B') then count else PROCS end < 1"),
        F.lit('a')
    ).otherwise(F.col('col'))
)
User contributions licensed under: CC BY-SA
2 People found this is helpful
Advertisement