Can I return pointer to VLA?

Question

Is such function prototype valid in C?

int (*func())[*];

And if it is, how can I define such functions?

Answer 1

You should return a pointer to an incomplete array type instead as * notation for variable-length arrays is only valid in parameter lists.

Example prototype and function definition:

extern float (*first_row(unsigned, unsigned, float (*)[*][*]))[];

float (*first_row(unsigned n, unsigned m, float (*matrix)[n][m]))[]
{
    return *matrix;
}

You'd invoke it like this:

unsigned n = 3, m = 4;
float matrix[n][m];
float (*row)[m] = first_row(n, m, &matrix);

Note that it is undefined behaviour to return a pointer to an array (variable-length or otherwise) that has been declared within the function if it has automatic storage duration. This implies that you can only return a pointer to a variable-length array that you passed in as an argument or allocated dynamically.