Skip to content
Advertisement

Transpose Columns having values into Rows

I have a requirement that needs columns with values to be transposed into rows. For instance refer to the table below:

cust:

Given the above table, we have to find out which columns have a value and if there is a value in that column then there should be a row created. For e.g.

For the customer 1234 only cover1 and cover 3 are populated hence there will be 2 records created. For 5678 cover1 & cover2 are Nulls hence, only 1 record for cover_3 needs to be created.

I could apply a simple approach like below. But I was wondering if there is an elegant approach and a smarter solution to this.

Please share your thoughts. We use Spark-SQL 2.4

Thanks

Advertisement

Answer

The requirement could be met by using Spark’s stack() with the %sql mode:

User contributions licensed under: CC BY-SA
6 People found this is helpful
Advertisement