Presently, we are living in a hyper-connected world where millions of heterogeneous
devices are continuously sharing information in different application contexts for wellness, improving
communications, digital businesses, etc. However, the bigger the number of devices and connections
are, the higher the risk of security threats in this scenario. To counteract against malicious behaviours
and preserve essential security services, Network Intrusion Detection Systems (NIDSs) are the
most widely used defence line in communications networks. Nevertheless, there is no standard
methodology to evaluate and fairly compare NIDSs. Most of the proposals elude mentioning
crucial steps regarding NIDSs validation that make their comparison hard or even impossible.
This work firstly includes a comprehensive study of recent NIDSs based on machine learning
approaches, concluding that almost all of them do not accomplish with what authors of this paper
consider mandatory steps for a reliable comparison and evaluation of NIDSs. Secondly, a structured
methodology is proposed and assessed on the UGR’16 dataset to test its suitability for addressing
network attack detection problems. The guideline and steps recommended will definitively help
the research community to fairly assess NIDSs, although the definitive framework is not a trivial
task and, therefore, some extra effort should still be made to improve its understandability and
usability further.
