.png)
A Hybrid TF-IDF and Knowledge Graph-Enhanced Retrieval-Augmented Generation Framework with Large Language Models for Domain-Aware Question Answering
Abstract
This study aims to develop a domain-aware legal Question-Answering (QA) system tailored for Indonesia’s Micro, Small, and Medium Enterprises (MSMEs) by proposing a hybrid Retrieval-Augmented Generation (RAG) framework that integrates Term Frequency–Inverse Document Frequency (TF-IDF), Knowledge Graph (KG), and Large Language Model (LLM) components. In this framework, TF-IDF contributes by performing lexical-level retrieval to identify the most relevant documents based on keyword weighting; the KG enriches this retrieval by providing semantic relationships among legal entities, enabling deeper contextual understanding; and the LLM generates coherent responses conditioned on both lexical and semantically grounded evidence. Together, these components work synergistically to strengthen factual grounding during retrieval and improve contextual reasoning during generation. Methodologically, the system processes a curated dataset of 1,400 legal question–answer pairs collected from national legal repositories, including legislation, government regulations, and MSME digitalization guidelines. The process includes text preprocessing, keyword extraction using TF-IDF, semantic enrichment through a KG that maps legal entities and their relationships, and answer generation via an LLM powered by the RAG pipeline. The system was evaluated using Precision, Recall, F1-Score, Bilingual Evaluation Understudy (BLEU), and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics, validated by five legal experts. Results show an accuracy improvement from 76.5% to 83.5% after integrating KG, with Precision of 0.853, Recall of 0.877, and F1-Score of 0.865. The generative evaluation yielded a BLEU score of 0.9276 and ROUGE-L of 0.9301, indicating strong linguistic and semantic alignment between system outputs and expert-authored references. The study concludes that this approach offers a practical foundation for building AI-based legal assistance tools and highlights future opportunities for expansion to other legal domains and multilingual RAG applications.
Keywords
Full Text:
PDFReferences
S. Ajay Mukund and K. S. Easwarakumar, “Optimizing Legal Text Summarization Through Dynamic Retrieval-Augmented Generation and Domain-Specific Adaptation,” Symmetry (Basel)., vol. 17, no. 5, 2025, doi: 10.3390/sym17050633.
W. Zhuohao, W. Dong, and L. Qing, “Keyword Extraction from Scientific Research Projects Based on SRP‐TF‐IDF,” Chinese J. Electron., vol. 30, no. 4, pp. 652–657, 2021.
I. Radeva, I. Popchev, L. Doukovska, and M. Dimitrova, “Web Application for Retrieval-Augmented Generation: Implementation and Testing,” Electron., vol. 13, no. 7, 2024, doi: 10.3390/electronics13071361.
N. A. Akbar, R. Dembani, B. Lenzitti, and D. Tegolo, “RAG-Driven Memory Architectures in Conversational LLMs—A Literature Review With Insights Into Emerging Agriculture Data Sharing,” IEEE Access, vol. 13, no. June, pp. 123855–123880, 2025, doi: 10.1109/ACCESS.2025.3589241.
L. C. Chen, M. S. Pardeshi, Y. X. Liao, and K. C. Pai, “Application of retrieval-augmented generation for interactive industrial knowledge management via a large language model,” Comput. Stand. Interfaces, vol. 94, no. September 2024, p. 103995, 2025, doi: 10.1016/j.csi.2025.103995.
L. Ching Chen, “An extended TF-IDF method for improving keyword extraction in traditional corpus- based research: An example of a climate change corpus,” Data Knowl. Eng., vol. 153, 2024, doi: https://doi.org/10.1016/j.datak.2024.102322.
Y. Nuri and E. Senyurek, “Filtering articles based on their abstracts using TF-IDF,” Int. J. Adv. Eng. Manag., vol. 6, no. 08, pp. 364–368, 2024, doi: 10.35629/5252-0608364368.
H. J. Kim, J. W. Baek, and K. Chung, “Optimization of associative knowledge graph using TF-IDF based ranking score,” Appl. Sci., vol. 10, no. 13, 2020, doi: 10.3390/app10134590.
Y. Wang, “Research on the TF–IDF algorithm combined with semantics for automatic extraction of keywords from network news texts,” J. Intell. Syst., vol. 33, no. 1, 2024, doi: 10.1515/jisys-2023-0300.
H. Liu, X. Chen, and X. Liu, “A Study of the Application of Weight Distributing Method Combining Sentiment Dictionary and TF-IDF for Text Sentiment Analysis,” IEEE Access, vol. 10, pp. 32280– 32289, 2022, doi: 10.1109/ACCESS.2022.3160172.
J. Zhou, Z. Ye, S. Zhang, Z. Geng, N. Han, and T. Yang, “Investigating response behavior through TF- IDF and Word2vec text analysis: A case study of PISA 2012 problem-solving process data,” Heliyon, vol. 10, no. 16, p. e35945, 2024, doi: 10.1016/j.heliyon.2024.e35945.
A. R. Lubis, M. K. M. Nasution, O. S. Sitompul, and E. M. Zamzami, “The effect of the TF-IDF algorithm in times series in forecasting word on social media,” Indones. J. Electr. Eng. Comput. Sci., vol. 22, no. 2, pp. 976–984, 2021, doi: 10.11591/ijeecs.v22.i2.pp976-984.
Y. Wang, D. Zhang, Y. Yuan, K. Liu, and Y. Yang, “Improvement of TF-IDF Algorithm Based on Knowledge Graph,” in 2018 IEEE 16th International Conference on Software Engineering Research, Management and Applications (SERA), 2018, pp. 19–24, doi: 10.1109/SERA.2018.8477196.
M. M. Hussien, A. N. Melo, A. L. Ballardini, C. S. Maldonado, R. Izquierdo, and M. Á. Sotelo, “RAG- based explainable prediction of road users behaviors for automated driving using knowledge graphs and large language models,” Expert Syst. Appl., vol. 265, no. July 2024, p. 125914, 2025, doi: 10.1016/j.eswa.2024.125914.
I. Harrando and R. Troncy, “Combining Semantic and Linguistic Representations for Media Recommendation,” HAL open Sci., 2022.
X. Kehan, Z. Kun, L. Jingyuan, and W. Yuanzhuo, “CRP-RAG: A Retrieval-Augmented Generation Framework for Supporting Complex Logical Reasoning and Knowledge Planning,” Electron., vol. 14, no. 1, pp. 1–34, 2025, doi: 10.3390/electronics14010047.
Y. Li, V. Zakhozhyi, D. Zhu, and L. J. Salazar, “Domain Specific Knowledge Graphs as a Service to the Public: Powering Social-Impact Funding in the US,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020, no. 1, pp. 2793–2801, doi: 10.1145/3394486.3403330.
Z. Wang, Z. Liu, W. Lu, and J. Lu, “Improving knowledge management in building engineering with hybrid retrieval-augmented generation framework,” J. Build. Eng., vol. 103, 2025, doi: https://doi.org/10.1016/j.jobe.2025.112189.
V. Armant et al., “Can Knowledge Graphs and Retrieval-Augmented Generation be combined to Explain Query / Answer Relationships Truthfully ? To cite this version : HAL Id : hal-04874325 Can Knowledge Graphs and Retrieval-Augmented Generation be combined to Explain Query / An,” 2025.
A. Zubiaga, “Natural language processing in the era of large language models,” Front. Artif. Intell., vol. 6, 2023, doi: 10.3389/frai.2023.1350306.
L. Bahr, C. Wehner, J. Wewerka, J. Bittencourt, U. Schmid, and R. Daub, “Knowledge graph enhanced retrieval-augmented generation for failure mode and effects analysis,” J. Ind. Inf. Integr., vol. 45, no. February, 2025, doi: 10.1016/j.jii.2025.100807.
Z. Li, Z. Wang, W. Wang, K. Hung, H. Xie, and F. L. Wang, “Retrieval-augmented generation for educational application: A systematic survey,” Comput. Educ. Artif. Intell., vol. 8, no. May, p. 100417, 2025, doi: 10.1016/j.caeai.2025.100417.
B. Han, T. Susnjak, and A. Mathrani, “Automating Systematic Literature Reviews with Retrieval- Augmented Generation: A Comprehensive Overview,” Appl. Sci., vol. 14, no. 19, 2024, doi: 10.3390/app14199103.
L. Xu, L. Lu, M. Liu, C. Song, and L. Wu, “Nanjing Yunjin intelligent question-answering system based on knowledge graphs and retrieval augmented generation technology,” Herit. Sci., vol. 12, no. 1, pp. 1–23, 2024, doi: 10.1186/s40494-024-01231-3.
A. O. M. Saleh, G. Tur, and Y. Saygin, “SG-RAG MOT : SubGraph Retrieval Augmented Generation with Merging and Ordering Triplets for Knowledge Graph Multi-Hop Question Answering,” pp. 1–24, 2025.
Y. Shang et al., “Empowering knowledge graphs with hybrid retrieval-augmented generation for the intelligent mix scheme of mass concrete,” Case Stud. Constr. Mater., vol. 23, no. May, p. e04979, 2025, doi: 10.1016/j.cscm.2025.e04979.
M. DeBellis, N. Dutta, G. Jacob, and A. Balaji, “Integrating Ontologies and Large Language Models to Implement Retrieval Augmented Generation,” Appl. Ontol., vol. 19, no. 4, pp. 389–407, 2025, doi: 0.1177/15705838241296446.
J. Liang et al., “GeoGraphRAG: A graph-based retrieval-augmented generation approach for empowering large language models in automated geospatial modeling,” Int. J. Appl. Earth Obs. Geoinf., vol. 142, no. June, p. 104712, 2025, doi: 10.1016/j.jag.2025.104712.
J. Miao, C. Thongprayoon, S. Suppadungsuk, O. A. Garcia Valencia, and W. Cheungpasitporn, “Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: Advancing Practical Applications,” Med., vol. 60, no. 3, pp. 1–15, 2024, doi: 10.3390/medicina60030445.
J. Brand, A. Israeli, and D. Ngwe, “Using LLMs for Market Research,” 2024.
A. Petukhova and N. Fachada, “TextCL: A Python package for NLP preprocessing tasks,” SoftwareX, vol. 19, 2022, doi: 10.1016/j.softx.2022.101122.
B. Probierz and J. Kozak, “Knowledge graphs to an analysis and visualization of texts from scientific articles,” Procedia Comput. Sci., vol. 225, pp. 4324–4333, 2023, doi: 10.1016/j.procs.2023.10.429.
A. Erfina and M. R. N. R. Alamsyah, “Implementation of Naive Bayes classification algorithm for Twitter user sentiment analysis on ChatGPT using Python programming language,” Data Metadata, vol. 2, pp. 2–11, 2023, doi: 10.56294/dm202345.
H. Mehta and K. Passi, “Social media hate speech detection using explainable AI,” Algorithms, vol. 15, no. 8, p. 291, 2022.
L. R. Halim and A. Suryadibrata, “Cyberbullying Sentiment Analysis with Word2Vec and One- Against-All Support Vector Machine,” Int. J. New Media Technol., vol. 8, no. 1, p. 57, 2021.
M. A. Palomino and F. Aider, “Evaluating-the-Effectiveness-of-Text-PreProcessing-in-Sentiment- AnalysisApplied-Sciences-Switzerland.pdf,” Mdpi, vol. 12, p. 8765, 2022.
A. Jabbar, M. I. Tamimy, A. Akhunzada, S. Iqbal, and S. Hussain, “Empirical evaluation and study of text stemming algorithms,” Artif. Intell. Rev., vol. 53, pp. 5559–5588, 2020, doi: https://doi.org/10.1007/s10462-020-09828-3.
A. Farhan AlShammari, “Implementation of Keyword Extraction using Term Frequency-Inverse Document Frequency (TF-IDF) in Python,” Int. J. Comput. Appl., vol. 185, no. 35, pp. 975–8887, 2023.
L. Zhang, “Features extraction based on Naive Bayes algorithm and TF-IDF for news classification,” PLoS One, vol. 20, no. 7 July, pp. 1–17, 2025, doi: 10.1371/journal.pone.0327347.
M. Atef Mosa, “Predicting Semantic Categories in Text Based on Knowledge Graph Combined with Machine Learning Techniques,” Appl. Artif. Intell., vol. 35, no. 12, pp. 933–951, 2021, doi: 10.1080/08839514.2021.1966883.
Q. Wu et al., “Integrating Knowledge Graph and Machine Learning Methods for Landslide Susceptibility Assessment,” Remote Sens., vol. 16, no. 13, 2024, doi: 10.3390/rs16132399.
A. Chen, Y. Tian, J. Zhang, C. Li, and H. Zhang, “LLM-based intelligent Q & A system for railway locomotive maintenance standardization,” 2025. doi: https://doi.org/10.1038/s41598-025-96130-3.
C. Gan, Q. Zhang, and T. Mori, “Application of LLM Agents in Recruitment : A Novel Framework for Automated Resume Screening,” J. Inf. Process., vol. 32, pp. 881–893, 2024, doi: 10.2197/ipsjjip.32.881.
W. Shi, W. Zheng, J. X. Yu, H. Cheng, and L. Zou, “Keyphrase Extraction Using Knowledge Graphs,” Data Sci. Eng., vol. 2, no. 4, pp. 275–288, 2017, doi: 10.1007/s41019-017-0055-z.
N. U. R. Izyan, Y. Saat, M. Mohd, S. Azman, M. Noah, and S. M. Al-ghuribi, “Beyond Relevance : Enhancing Serendipity in Content-Based Recommendations With Knowledge Graphs,” IEEE Access, vol. 13, no. August, pp. 142980–142989, 2025, doi: 10.1109/ACCESS.2025.3598342.
C. Peng, F. Xia, M. Naseriparsa, and F. Osborne, “Knowledge Graphs : Opportunities and Challenges,” Artif. Intell. Rev., vol. 56, no. 11, pp. 13071–13102, 2023, doi: 10.1007/s10462-023-10465-9.
K. Hou, J. Li, Y. Liu, S. Sun, H. Zhang, and H. Jiang, “KG-EGV: A Framework for Question Answering with Integrated Knowledge Graphs and Large Language Models,” Electronics, vol. 13, no. 23, pp. 1–23, 2024, doi: 10.3390/electronics13234835.
DOI: https://doi.org/10.47738/jads.v7i2.1136
Refbacks
- There are currently no refbacks.
Journal of Applied Data Sciences
| ISSN | : | 2723-6471 (Online) |
| Publisher | : | Bright Publisher |
| Website | : | http://bright-journal.org/JADS |
| : | taqwa@amikompurwokerto.ac.id (principal contact) | |
| support@bright-journal.org (technical issues) |
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0