Investigating Knowledge Graphs for Context-Aware Search in Financial Transactional AI Agents
No Thumbnail Available
Date
2026
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
SDU University
Abstract
This thesis presents an empirical evaluation of a hybrid knowledge-graph and largelanguage-model (KG–LLM) architecture for financial transactional compliance reasoning. The architecture was proposed in a companion conference paper [1] that established the design and demonstrated it on a small worked example; the contribution of this thesis is the quantitative, ablation-based, robustness-aware empirical study that completes the proposal. A deterministic synthetic dataset of 500 transactions across eight compliance-rule scenarios plus a clean negative class is generated under two complementary disciplines: a crossrule labeling pass that mirrors how real anti-money-laundering pipelines assign overall compliance, and an explicit per-channel audit of base-context fields for label-correlated content (which uncovered, and fixed, a substantial description-field scenario-leakage problem in the dataset’s earlier versions; the empirical impact of the leakage and the corrected results are reported throughout). A four-arm ablation matrix isolates the contribution of structured KG relations by varying only the grounding mode supplied to the language model. A parametric fifth arm sweeps per-transaction KG coverage from 100 % down to 0 % to characterise the architecture’s degradation under realistic incomplete-graph conditions. On the corrected, description-neutral dataset, the KG-augmented arm achieves accuracy 0.891 with perfect precision (1.000, zero wrongly-cleared violations across 1500 predictions), recall 0.691, and F1 0.817 across three independent repeats. Both no-KG baselines—a vanilla LLM and a rule-text-grounded LLM—collapse to majority-class-default behaviour at F1 of approximately 0.01, so the architecture roughly fifty-fold lifts F1 relative to the baselines while never wrongly clearing a violation. The KG-augmented arm’s accuracy advantage degrades smoothly with KG coverage and disappears at coverage below approximately 20 % (the dropout sweep crosses the no-KG baseline at p ∗ ≈ 0.81). The methodological contributions of the work include the rule-name neutrality discipline—a previously-undocumented prompt-design pitfall whereby loaded rule identifiers override structured per-rule relations in language-model reasoning, identified during the project’s first benchmark and described in detail as finding F-001 in the project record—and the description-channel audit protocol (F-007) that the thesis argues should be a default hygiene step in any synthetic-data pipeline feeding a language model.
Description
Keywords
knowledge graphs, large language models, retrieval-augmented generation, financial AI agents
Citation
Baisenkul N / Investigating Knowledge Graphs for Context-Aware Search in Financial Transactional AI Agents / SDU University / Department of Computer Science