Most efficient row multiplication with matrix in Pandas

Question

Suppose I have a matrix as such

df = pd.DataFrame(randint(2,size=(3,9)))
df.values
array([[0, 1, 0, 1, 1, 1, 0, 1, 1],
       [1, 0, 1, 1, 1, 1, 0, 0, 1],
       [0, 0, 0, 1, 0, 0, 1, 1, 0]])

Again; each row in this example represents three 3D coordinates, that need to rotated by, e.g. the following rotation matrix:

array([[  0.00000000e+00,   0.00000000e+00,   1.00000000e+00],
       [  0.00000000e+00,   1.00000000e+00,   0.00000000e+00],
       [ -1.00000000e+00,   0.00000000e+00,   0.00000000e+00]])

To do this as efficiently as possible (the real problem has millions of coordinates btw) currently, I am somewhat baffled that I have to do:

First apply df.reshape - each row in this example consists of three 3D coordinates as so [(x,y,z),(x,y,z),(x,y,z)]:

array([[0, 1, 0],
       [1, 1, 1],
       [0, 1, 1],
       [1, 0, 1],
       [1, 1, 1],
       [0, 0, 1],
       [0, 0, 0],
       [1, 0, 0],
       [1, 1, 0]])

Then in order to get it to rotate to convention, one must take u_new = R \dot u which means the transpose of the above, so that we can take a column-wise (i.e. coordinate) multiplication with the rotation matrix.

array([[0, 1, 0, 1, 1, 0, 0, 1, 1],
       [1, 1, 1, 0, 1, 0, 0, 0, 1],
       [0, 1, 1, 1, 1, 1, 0, 0, 0]])

Then we can do the multiplication:

pd.DataFrame(dot(rotmat,df)).values

array([[  0.00e+00,   2.22e-16,   0.00e+00,   1.00e+00,   2.22e-16,
          2.22e-16,   1.00e+00,   1.00e+00,   2.22e-16],
       [  1.00e+00,   0.00e+00,   1.00e+00,   1.00e+00,   1.00e+00,
          1.00e+00,   0.00e+00,   0.00e+00,   1.00e+00],
       [  0.00e+00,  -1.00e+00,   0.00e+00,  -1.00e+00,  -1.00e+00,
         -1.00e+00,   2.22e-16,  -1.00e+00,  -1.00e+00]])

And then reverse the whole process to get this back into the original shape, to be used for other purposes.

Surely there must be more efficient ways to do this (hopefully without messing with the rotation matrix)?

Answer 1

This should never touch a dataframe until you are done with your transformations.

a = np.array([
        [0, 1, 0, 1, 1, 1, 0, 1, 1],
        [1, 0, 1, 1, 1, 1, 0, 0, 1],
        [0, 0, 0, 1, 0, 0, 1, 1, 0]
    ])

rotmat = np.array([
        [  0.00000000e+00,   0.00000000e+00,   1.00000000e+00],
        [  0.00000000e+00,   1.00000000e+00,   0.00000000e+00],
        [ -1.00000000e+00,   0.00000000e+00,   0.00000000e+00]
    ])

a.reshape(3, 3, -1).dot(rotmat).reshape(-1, 9)

array([[ 0.,  1.,  0., -1.,  1.,  1., -1.,  1.,  0.],
       [-1.,  0.,  1., -1.,  1.,  1., -1.,  0.,  0.],
       [ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  1.,  1.]])

---

df = pd.DataFrame(a.reshape(3, 3, -1).dot(rotmat).reshape(-1, 9))
df