Evaluating NLP tools for AI in software requirements analysis

Abstract

Software requirements analysis is increasingly automated by applying natural language processing (NLP) tools, enhancing efficiency and precision. This research employs the Mendeley FR_NFR dataset to evaluate the classification of functional requirements (FR) and non-functional requirements (NFR) utilising three NLP tools: NLTK, OpenAI, and spaCy. The evaluation uses performance indicators like F1-score, recall, accuracy, precision, and confusion matrices. OpenAI is a good option for high-stakes applications because of its 94% F1 score and exceptional accuracy, even with the related API expenses. With 83% accuracy and 0.1 s per query, SpaCy is ideal for real-time applications because it balances speed and efficiency. With its 68% accuracy rate, NLTK’s rule-based methodology is still a viable choice for prototyping or in controlled settings where transparency is crucial. With an average accuracy of 92%, the results show that OpenAI’s transformer-based model performs better than NLTK and spaCy, even though spaCy has an advantage in entity recognition. This study provides practitioners with critical insights by elucidating the trade-offs between accuracy, interpretability, and computational efficiency.