I understood the problem statement to take the transcriptions and store the embeddings in the database. I would like to contribute to this issue . Please assign it to me!!

I have good knowledge of working on NLP and I also understand your problem . So I would to contribute to this issue . Could you please assign it to me

I have worked on a similar problem statement earlier. We had been given paragraphs on several topics and then a question on a specific topic was asked and we had to retrieve the answer for that query using the given paragraphs. The solution we had come up with was to convert the paragraphs into embeddings using the hugging face transformer model. The embeddings were indexed using the FAISS indexing library. Then for a question, we took its embeddings and retrieved the closest paragraph embeddings from the index using cosine similarity. Here is the link to the code notebook for reference click here

We used retrieval then question Answering to solve the problem.

Thus, I think I can work on converting the above code into a proper API as required by the project.

I have made FastAPI to upload the PDF file and extract the text as per mentioned in the "git with basic implementation". I implemented the requirements of "Behavior of Upload API". Please review it

The approach which initally suggested was to creating a window for the embeddings and checking for any sharp changes in the embeddings.

However, now we aren't sure if the changing in the similarity score is a good enough approach as information about a variety of things may be present in a paragraph and this then separates them into different chunks.

I think what will be required will be :
A benchmark model using GPT that takes all the text in a page and creates chunks out of it.
Explore topic modelling style approach to the problem that does some kind of heirarchial clustering a page into identified topics.

Some sample PDFs are provided here. A simple test can be done on a page and we can see if the text extracted is getting chunked into the same paragraphs as in the pdf.

Okay sir, Currently I am dividing the page into chunks and then I am doing embedding. So now what I need to do is first divide the content of pages based on different topic and then do the embedding. Have I understood right Sir?

I will explore the PDFs you have attached.

Okay sir, Currently I am dividing the page into chunks and then I am doing embedding. So now what I need to do is first divide the content of pages based on different topic and then do the embedding. Have I understood right Sir?

I will explore the PDFs you have attached.

That is correct.

Potential flow for solving this could be :

• Pick a pdf - a good example seems to be https://drive.google.com/drive/u/0/folders/1sAsuh-EFH-xmFYrxzhmj0VRUZYNzsyLw from the pdfs provided in the folders.
• Decide an evaluation metric. For the above pdf, the text has neat paragrpahs that can be considered chunks. Create a test set- with the pdf parsed according to the headings to get the paragraphs.
• Use various methods to extract text from pdf and chunk it into various paragraphs such that are of similar topic
• Measure the accuracy of the chunking for various chunking methods by comparing your chunks vs the pdf paragraphs.
• Store the chunks in a CSV/DB.

Next steps:

• Generate tags for each chunk that can be searched for various user questions/prompts.
• Embed the tags using any vecotr embeddings
• Integrate the setup within a vector DB

Ok sir

I have been working on this project since 20 May. I did see many projects, but finally, I will understand everything related to this problem statement. I am submitting only this proposal. I have good knowledge of NLP and have been learning about HuggingFace Transformers for the last 1 week. I am interested in this project. I have been doing Machine learning for the last 1 year, so I have good knowledge of Python Language.

I have worked on more or else similar project before using hugging face transformer model. I would like to contribute on this project.

Hello @GautamR-Samagra Sir, I wanted to contribute to the development of the document uploader API within the AI Toolchain, for
helping streamline document processing, embedding generation, and indexing for enhanced machine learning workflows.