Converting 2D numpy array to 3D array without looping

Question

I have a 2D array of shape (t*40,6) which I want to convert into a 3D array of shape (t,40,5) for the LSTM's input data layer. The description on how the conversion is desired in shown in the figure below. Here, F1..5 are the 5 input features, T1...40 are the time steps for LSTM and C1...t are the various training examples. Basically, for each unique "Ct", I want a "T X F" 2D array, and concatenate all along the 3rd dimension. I do not mind losing the value of "Ct" as long as each Ct is in a different dimension.

I have the following code to do this by looping over each unique Ct, and appending the "T X F" 2D arrays in 3rd dimension.

# load 2d data
data = pd.read_csv('LSTMTrainingData.csv')

trainX = []

# loop over each unique ct and append the 2D subset in the 3rd dimension
for index, ct in enumerate(data.ct.unique()):
    trainX.append(data[data['ct'] == ct].iloc[:, 1:])

However, there are over 1,800,000 such Ct's so this makes it quite slow to loop over each unique Ct. Looking for suggestions on doing this operation faster.

EDIT:

data_3d = array.reshape(t,40,6)
trainX = data_3d[:,:,1:]

This is the solution for the original question posted.

Updating the question with an additional problem: the T1...40 time steps can have the highest number of steps = 40, but it could be less than 40 as well. The rest of the values can be 'np.nan' out of the 40 slots available.

Answer 1

Since all Ct have not the same length , you have no other choice than rebuild a new block. But use of data[data['ct'] == ct] can be O(n²) so it's a bad way to do it.

Here a solution using Panel . cumcount renumber each Ct line :

t=5
CFt=randint(0,t,(40*t,6)).astype(float) # 2D data
df= pd.DataFrame(CFt)
df2=df.set_index([df[0],df.groupby(0).cumcount()]).sort_index()
df3=df2.to_panel()

This automatically fills missing data with Nan. But It warns :

DeprecationWarning: Panel is deprecated and will be removed in a future version. The recommended way to represent these types of 3-dimensional data are with a MultiIndex on a DataFrame, via the Panel.to_frame() method

So perhaps working with df2 is the recommended way to manage your data.