Functional Components Description
First, there are several databases. The main one, called entity database, stores all the entity records. There is also a separate database to store original data from primary sources (events, alerts). These may be actually multiple databases, since different technologies may be suitable for different types of data. At last, there is an instance of a fast in-memory key-value database (Redis), that serves for storage of various, often temporary or fast changing, data that need to be accessed globally by different components. It also serves for various logging and caching purposes.
The main input of the system is represented by a set of primary data receivers. They receive messages, alerts, or entity lists from the primary data sources and put tasks (called update requests) to a global queue to create or update records of related entities. The tasks are processed by the core of the system – a set of workers. They apply the requested updates on given entity records. Also, the workers fetch data from external (secondary) sources and compute other attributes. This functionality is handled by plug-in modules, so it is easy to add, change or remove secondary data sources or computed attributes.
The workers may run in any number of instances in parallel, which makes the system easily scalable. Tasks are distributed to workers according to a hash of the entity identifier (a task always involves processing of a single entity record only), so the same record is always handled by the same worker. This helps to avoid a need for record locking and other concurrency issues. If the processing core or a plug-in needs to store a global state information, it is stored outside the workers, usually in the Redis database. Most of the plug-ins fetches data from external sources. Depending on the type and availability of the data, three methods of data acquisition are used: (i) the data are queried directly from their original sources by the plug-in module (for example, whois data are got this way); (ii) there is a special microservice or a cache to provide easier or more efficient access to the data (an example Services provided From a user’s point of view, the Network Entity Reputation Database (NERD) system is an online portal where the user can search any IP address, domain name or another network identifier (an entity) and get all security related information known about it list of all alerts that reported it as a source of some malicious activity, whether it is listed on some blacklists or other databases, related information from DNS, whois, geolocation, or data from internet wide scanning services. It is also possible to search for entities that match various criteria and sort them by various attributes or by a score summarizing the associated threat level. There is also a REST API for easy integration of the data into any other security or threat intelligence system. Behind the web portal and the API, which servers to access the data, there is a complex, modular system to acquire and process data from various sources, store them to the database and periodically update them.
Network entity reputation database
Virtual server WMware
8x vCPU (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz)
32 GB RAM