Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can we add a list of documents to an existing index in llama-index?

I have an existing index that is created using GPTVectorStoreIndex. However, when I am trying to add a new document to the existing index using the insert method, I am getting the following error :

AttributeError: 'list' object has no attribute 'get_text'

my code for updating the index is as follows :

max_input_size = 4096
num_outputs = 5000
max_chunk_overlap = 256
chunk_size_limit = 3900
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=num_outputs))
    
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper)

directory_path = "./trial_docs"
file_metadata = lambda x : {"filename": x}
reader = SimpleDirectoryReader(directory_path, file_metadata=file_metadata)
    
documents = reader.load_data()
print(type(documents))
index.insert(document = documents, service_context = service_context)
like image 487
Vivek Avatar asked Oct 15 '25 09:10

Vivek


1 Answers

I got it right, the mistake I was doing it was passing documents as a whole, which is a List object. The right way to update is as follows

max_input_size = 4096
num_outputs = 5000
max_chunk_overlap = 256
chunk_size_limit = 3900
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=num_outputs))
    
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper)

directory_path = "./trial_docs"
file_metadata = lambda x : {"filename": x}
reader = SimpleDirectoryReader(directory_path, file_metadata=file_metadata)
    
documents = reader.load_data()
print(type(documents))
for d in documents:
    index.insert(document = d, service_context = service_context)
like image 61
Vivek Avatar answered Oct 18 '25 00:10

Vivek



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!