How much does artificial intelligence know about Colombian law?

This article evaluates the ability of three large language models (LLMs)-ChatGPT 3.5, ChatGPT 4, and Gemini Pro-to answer legal queries in the Colombian context. A thorough analysis of responses to questions in five areas of law found that while ChatGPT 4 obtained the best overall results, none of the models reached an acceptable level of accuracy for practical application in the legal field. The results suggest that, although LLMs offer great potential to automate legal tasks, their use in practice requires rigorous human supervision and continuous development of the models. This study contributes to the debate on the ethics and legal implications of artificial intelligence in the legal field, highlighting the need to establish appropriate regulatory frameworks to ensure responsible use of these technologies.