IEEE SmartData 2024 Conference paper

Assessing the Syllogistic Logic and Fact-Checking Capabilities of Large Language Models

Cecilia Delgado Solorzano, Manuel Delaflor, Carlos Toxtli-Hernández

2024 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics, 2024 · pp. 479-488

DOI Website

Abstract

This paper presents an analysis of the logical and fact-checking capabilities of Large Language Models (LLMs) in the context of syllogistic reasoning. The study evaluates both proprietary and open-source LLMs using a test suite of 256 syllogisms. Human participants provide a baseline for comparison. The results reveal that a few proprietary LLMs demonstrate superior performance in identifying valid syllogisms compared to open-source models and human participants. However, human participants exhibit unique fact-checking capabilities when dealing with ambiguous or nuanced information. The discussion explores the implications of these findings, highlighting the need for more sophisticated evaluation methods. This study contributes to the ongoing discourse on the logical and fact-checking capabilities of LLMs and provides insights into their current limitations and future development.

Cite this work

Cecilia Delgado Solorzano, Manuel Delaflor, and Carlos Toxtli-Hernández. 2024. Assessing the Syllogistic Logic and Fact-Checking Capabilities of Large Language Models. 2024 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics. https://doi.org/10.1109/ithings-greencom-cpscom-smartdata-cybermatics62450.2024.00094

@inproceedings{Delgado2024Assessing,
  title = {Assessing the Syllogistic Logic and Fact-Checking Capabilities of Large Language Models},
  url = {http://dx.doi.org/10.1109/iThings-GreenCom-CPSCom-SmartData-Cybermatics62450.2024.00094},
  doi = {10.1109/ithings-greencom-cpscom-smartdata-cybermatics62450.2024.00094},
  booktitle = {2024 IEEE International Conferences on Internet of Things (iThings) and IEEE Green Computing & Communications (GreenCom) and IEEE Cyber, Physical & Social Computing (CPSCom) and IEEE Smart Data (SmartData) and IEEE Congress on Cybermatics},
  publisher = {IEEE},
  author = {Delgado-Solorzano, Cecilia and DelaFlor, Manuel and Toxtli, Carlos},
  year = {2024},
  month = Aug,
  pages = {479--488}
}

Related

Related publications