Importing a common constant declared pig file in other pig files

Question

Objective : Have defined constants (%declare and %default) statements in constants.pig for code modularity and to import the same in other pig files.

As per docs : http://pig.apache.org/docs/r0.12.0/cont.html#import-macros, %declare and %default are valid statements in a macro.

Issue Faced : Pig is not able to find the declared parameter.

Pig File : constants.pig

 %declare ACTIVE_VALUES 'UK';

Pig File : a.pig

 IMPORT 'constants.pig';

 A = LOAD 'a.csv' using PigStorage(',') AS (country_code:chararray, country_name:chararray);
 B = FILTER A BY country_code == '$ACTIVE_VALUES';
 dump B;

Input : a.csv

IN,India
US,United States
UK,United Kingdom

Error

Error before Pig is launched
----------------------------
ERROR 2997: Encountered IOException.      org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : ACTIVE_VALUES

 java.io.IOException: org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : ACTIVE_VALUES
at org.apache.pig.impl.PigContext.doParamSubstitution(PigContext.java:414)
at org.apache.pig.Main.runParamPreprocessor(Main.java:810)
at org.apache.pig.Main.run(Main.java:588)
at org.apache.pig.Main.main(Main.java:170)
Caused by: org.apache.pig.tools.parameters.ParameterSubstitutionException: Undefined parameter : ACTIVE_VALUES
at org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:355)
at org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:303)
at org.apache.pig.tools.parameters.PigFileParser.input(PigFileParser.java:67)
at org.apache.pig.tools.parameters.PigFileParser.Parse(PigFileParser.java:43)
at org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:95)
at org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:76)
at org.apache.pig.impl.PigContext.doParamSubstitution(PigContext.java:410)
... 3 more

My understanding of IMPORT is that the content of the imported pig will be executed and available from the calling pig script. If this is the case, the declared parameter should be available in the importing pig file.

Any inputs/ thoughts on having a common pig script file which will have declaration of constants and importing the same in other pig files to achieve code modularity.

Update :

A JIRA issue has already been raised on this. Ref. below links for details

1. https://issues.apache.org/jira/browse/PIG-2469?jql=text%20~%20%22macro%20%25default%22
2. http://grokbase.com/t/pig/user/121c685c55/error-using-define-in-a-macro
3. http://mail-archives.apache.org/mod_mbox/pig-user/201201.mbox/%3CCAB-acjN+hNAZn3Aws5usHW+At9rk=oFtyb26GxvKxBkjYNAODg@mail.gmail.com%3E

Answer 1

The IMPORT keyword is used to import macros, not constants. %declare and %default are preprocessor statements, and its scope is all the remaining lines in the script. If you declare it in a script, but import it from a different one, it will not work because it is out of scope.

Both statements are valid in a macro, as long as you use the declared variable inside the macro. If you need to define constants outside the script for modularity, you need to use a parameter file:

ACTIVE_VALUES = 'UK'

And then run your Pig script like the following:

pig -param_file your_params_file.properties -f your_script.pig

If you really want to use IMPORT, you could create a macro which takes care of the filtering with that constant value:

%declare ACTIVE_VALUES 'UK';

DEFINE my_custom_filter(A) RETURNS B {
   $B = FILTER $A BY $0 == '$ACTIVE_VALUES ';
};

And then import it like you were doing in your script but instead of calling the FILTER function, call your own macro:

IMPORT 'macro.pig';

A = LOAD 'a.csv' using PigStorage(',') AS (country_code:chararray, country_name:chararray);
B = my_custom_filter(A);
dump B;