Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

In Flink, how can I access the Key when applying a Process Function on a Keyed Stream?

In Flink, I have a keyed stream to which I am applying a Process Function.

myDataStream
  .keyBy(new MyKeySelector())
  .process(new FooBarProcessFunction())

My Key Selector looks something like this...

public class MyKeySelector implements KeySelector<FooBar, FooKey>

public FooKey getKey (FooBar value) {
   return new FooKey (value);
}

And FooBarProcessFunction looks something like this...

public class FooBarProcessFunction extends ProcessFunction<FooBar, TransformedFooBar> {

    public void processElement(FooBar newFooBar, Context ctx, Collector<FooBar> out) {
        //do something with newFooBar
        // *****but I also want to know the Key (FooKey) here***** 
    }
}

In FooBarProcessFunction, I would like to obtain the Key which was created by MyKeySelector's getKey method. Is that doable?

At present, I am using a workaround wherein I essentially recreate the Key in the processElement function. But it would be ideal if I can avoid doing so.

like image 849
victtim Avatar asked Oct 20 '25 10:10

victtim


1 Answers

In order to access to the key from a process function, you should use KeyedProcessFunction

Your example become:

public class FooBarProcessFunction extends KeyedProcessFunction<FooKey, FooBar, TransformedFooBar> {

    public void processElement(FooBar newFooBar, Context ctx, Collector<FooBar> out) {
        //do something with newFooBar
        // *****but I also want to know the Key (FooKey) here***** 
        ctx.getCurrentKey
    }
}
like image 94
theShadow89 Avatar answered Oct 22 '25 04:10

theShadow89



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!