Using Agent-Based Simulations to Evaluate Bayesian Networks for Criminal Scenarios

Ludi van Leeuwen, Bart Verheij, Rineke Verbrugge, Silja Renooij

Scenario-based Bayesian networks (BNs) have been proposed as a tool for the rational handling of evidence. The proper evaluation of existing methods requires access to a ground truth that can be used to test the quality and usefulness of a BN model of a crime. However, that would require a full probability distribution over all relevant variables used in the model, which is in practice not available. In this paper, we use an agent-based simulation as a proxy for the ground truth for the evaluation of BN models as tools for the rational handling of evidence. We use fictional crime scenarios as a background. First, we design manually constructed BNs using existing design methods in order to model example crime scenarios. Second, we build an agent-based simulation covering the scenarios of criminal and non-criminal behavior. Third, we algorithmically determine BNs using statistics collected experimentally from the agent-based simulation that represents the ground truth. Finally, we compare the manual, scenario-based BNs to the algorithmic BNs by comparing the posterior probability distribution over outcomes of the network to the ground-truth frequency distribution over those outcomes in the simulation, across all evidence valuations. We find that both manual BNs and algorithmic BNs perform similarly well: they are good reflections of the ground truth in most of the evidence valuations. Using ABMs as a ground truth can be a tool to investigate Bayesian Networks and their design methods, especially under circumstances that are implausible in real-life criminal cases, such as full probabilistic information.

Manuscript (in PDF-format)

Reference:
van Leeuwen, L., Verheij, B., Verbrugge, R., & Renooij, S. (2023). Using Agent-Based Simulations to Evaluate Bayesian Networks for Criminal Scenarios. The 19th International Conference on Artificial Intelligence and Law (ICAIL 2023). Proceedings of the Conference, 323-332. New York (New York): ACM. https://doi.org/10.1145/3594536.3595125

Bart Verheij's home page - research - publications