The idea for this project came from a real-world problem. My brother completed his MBBS and was preparing for the Foreign Medical Graduate Examination (FMGE). During his preparation, he had to go through a large number of previous questions, identify frequently asked topics, and revise concepts across different medical subjects. I wanted to explore whether Generative AI could make this process more efficient.
I started by building a pipeline to collect publicly available FMGE questions and topics and connected this data with a local Large Language Model (LLM). The goal was to create a study assistant where a student could ask questions, find relevant topics, and get explanations.
However, the initial LLM-based approach had two major problems: hallucination and outdated knowledge. The model could generate an answer that sounded correct even when it was not supported by the collected material. Also, whenever new questions or information were added, the LLM did not automatically have access to them.
This led me to implement Retrieval-Augmented Generation (RAG).
In the RAG architecture, the collected questions and topics are cleaned, divided into smaller chunks, converted into embeddings, and stored in a vector database. When a student asks a question, the system performs a semantic search to find the most relevant information and passes that context to the local LLM.
The workflow looks like this:
FMGE Data → Cleaning & Chunking → Embeddings → Vector Database
User Question → Semantic Search → Relevant Content → Local LLM → Answer
The major advantage of this approach is that the LLM no longer has to depend entirely on its internal knowledge. It can generate answers using information retrieved from the FMGE knowledge base. New questions can also be added to the database without retraining the entire model.
RAG does not completely eliminate hallucination, but it helps make responses more grounded, relevant, traceable, and easier to keep updated.
This project helped me understand an important lesson about Generative AI: sometimes improving an AI application is not about using a bigger model—it is about giving the model the right information at the right time.
Conclusion
This project showed me how RAG can overcome some of the practical limitations of standalone LLMs, especially hallucination and outdated knowledge. By combining a local LLM with a continuously updatable knowledge base, I was able to create a more reliable and relevant FMGE preparation assistant. More importantly, the project helped me understand that the effectiveness of an AI application depends not only on the model itself, but also on how the right data is retrieved and provided to the model at the right time.