Re-shape pandas dataframe stack/unstack

Question

df = pd.DataFrame({'BORDER':['GERMANY','FRANCE','ITALY','USA','CANADA','MEXICO','INDIA','CHINA','JAPAN' ], 'ASID':[21, 32, 99, 77,66,55,44,88,111], 'HOUR1':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6],'HOUR2':[3 ,3 ,3, 5 ,5 ,5, 7, 7, 7], 'HOUR3':[8 ,8 ,8, 12 ,12 ,12, 99, 99, 99], 'PRICE1':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6], 'PRICE2':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6],'PRICE3':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6] })

df = df[['ASID', 'BORDER', 'HOUR1', 'PRICE1', 'HOUR2', 'PRICE2', 'HOUR3', 'PRICE3']]

I've been trying to re-shape this dataframe for the past day. Tinkering with stack/unstack/melt and shifting columns into indicies etc, but have not been able to achieve my goal.

The desired output has the following columns:

ASID, BORDER, HOUR, PRICE

I want to stack all of ['HOUR1', 'HOUR2', HOUR3'] into a single column = HOUR.

Similarly, I want to stack all of ['PRICE1', 'PRICE2', 'PRICE3'] in a single column = PRICE, such that the value in this field is aligned with the corresponding value in the HOUR column. There is a link between HOUR1 & PRICE1, HOUR2 & PRICE2, HOUR3 & PRICE3.

I appreciate any guidance you can provide.

Answer 1

Original data (notice correction of 'PRICE1' on second row).

df = pd.DataFrame({'BORDER':['GERMANY','FRANCE','ITALY','USA','CANADA','MEXICO','INDIA','CHINA','JAPAN' ], 'ASID':[21, 32, 99, 77,66,55,44,88,111], 'HOUR1':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6],'HOUR2':[3 ,3 ,3, 5 ,5 ,5, 7, 7, 7], 'HOUR3':[8 ,8 ,8, 12 ,12 ,12, 99, 99, 99], 'PRICE1':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6], 'PRICE2':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6],'PRICE3':[2 ,2 ,2 ,4 ,4 ,4 ,6 ,6, 6] })

df = df[['ASID', 'BORDER', 'HOUR1', 'PRICE1', 'HOUR2', 'PRICE2', 'HOUR3', 'PRICE3']]

First, set the index to ASID and BORDER.

df.set_index(['ASID', 'BORDER'], inplace=True)

Then, create two DataFrames for prices and hours, stacking the results. Drop the hours and prices levels from these stacked DataFrames.

prices = df[['PRICE1','PRICE2', 'PRICE3']].stack()
prices.index = prices.index.droplevel(2)
hours = df[['HOUR1', 'HOUR2', 'HOUR3']].stack()
hours.index = hours.index.droplevel(2)

Finally, concatenate these two DataFrames and rename your columns.

df_new = pd.concat([hours, prices], axis=1)
df_new.columns = ['HOUR', 'PRICE']

>>> df_new
              HOUR  PRICE
ASID BORDER              
21   GERMANY     2      2
     GERMANY     3      2
     GERMANY     8      2
32   FRANCE      2      2
     FRANCE      3      2
     FRANCE      8      2
99   ITALY       2      2
     ITALY       3      2
     ITALY       8      2
77   USA         4      4
     USA         5      4
     USA        12      4
66   CANADA      4      4
     CANADA      5      4
     CANADA     12      4
55   MEXICO      4      4
     MEXICO      5      4
     MEXICO     12      4
44   INDIA       6      6
     INDIA       7      6
     INDIA      99      6
88   CHINA       6      6
     CHINA       7      6
     CHINA      99      6
111  JAPAN       6      6
     JAPAN       7      6
     JAPAN      99      6