Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Maven cannot find symbol, but independent build works

I'm writing a Pig UDF, and trying to use Maven to compile it into a jar. The code registers as being fine in Eclipse, but when I go to build it in Maven, I get a cannot find symbol error, shown below. I've checked the method signature for the openReader method, and it matches the classes that I'm passing it, so I'm not really sure what's going wrong with the Maven compilation.

Dependencies Snippet:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>${hadoop.version}</version>
    <exclusions>
        <exclusion>
            <groupId>hsqldb</groupId>
            <artifactId>hsqldb</artifactId>
        </exclusion>
        <exclusion>
            <groupId>net.sf.kosmosfs</groupId>
            <artifactId>kfs</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.eclipse.jdt</groupId>
            <artifactId>core</artifactId>
        </exclusion>
        <exclusion>
            <groupId>net.java.dev.jets3t</groupId>
            <artifactId>jets3t</artifactId>
        </exclusion>
        <exclusion>
            <groupId>oro</groupId>
            <artifactId>oro</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-mapred</artifactId>
    <version>1.7.5</version>
</dependency>
<dependency>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro</artifactId>
    <version>1.7.5</version>
</dependency>

Code Snippet:

import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.FileReader;
import org.apache.avro.file.SeekableInput;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumReader;
import org.apache.avro.mapred.FsInput;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;

protected String getLastRecord(String location) throws IOException {
  Path path = new Path(location);
  Configuration config = new Configuration();
  SeekableInput input = new FsInput(path, config);
  DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>();
  FileReader<GenericRecord> fileReader = DataFileReader.openReader(input, reader);
  Iterator<GenericRecord> itr = fileReader.iterator(); 

  long initialPosition = input.tell();
  itr.next();
  long newPosition = input.tell();

  input.seek(input.length() - (initialPosition - newPosition));
  return itr.next().get("id").toString();
};

Error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project newscred-recsys-udfs: Compilation failure
[ERROR] /Users/andrew/Code/recommender/hadoop/newscred_recsys/udfs/java/src/main/java/com/mortardata/newscred/recsys/SqoopAvroLoader.java:[104,64] cannot find symbol
[ERROR] symbol:   method openReader(org.apache.avro.file.SeekableInput,org.apache.avro.io.DatumReader<org.apache.avro.generic.GenericRecord>)
[ERROR] location: class org.apache.avro.file.DataFileReader
[ERROR] -> [Help 1]
like image 753
Andrew FigPope Avatar asked Nov 20 '25 16:11

Andrew FigPope


2 Answers

The project should be cleaned. Project -> Clean

like image 172
Sujith Mohan Avatar answered Nov 22 '25 06:11

Sujith Mohan


Apparently, the jar for org.apache.avro has a different set of methods than the jar distributed from the Apache website, causing the mismatch.

like image 32
Andrew FigPope Avatar answered Nov 22 '25 07:11

Andrew FigPope



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!