Adding a New Rule Type

This document describes how to create a new rule type. Built in rule types live in elastalert/ruletypes.py and are subclasses of RuleType. At the minimum, your rule needs to implement add_data.

Your class may implement several functions from RuleType:

class AwesomeNewRule(RuleType):
    # ...
    def add_data(self, data):
        # ...
    def get_match_str(self, match):
        # ...
    def garbage_collect(self, timestamp):
        # ...

You can import new rule types by specifying the type as module.file.RuleName, where module is the name of a Python module, or folder containing __init__.py, and file is the name of the Python file containing a RuleType subclass named RuleName.

Basics

The RuleType instance remains in memory while ElastAlert is running, receives data, keeps track of its state, and generates matches. Several important member properties are created in the __init__ method of RuleType:

self.rules: This dictionary is loaded from the rule configuration file. If there is a timeframe configuration option, this will be automatically converted to a datetime.timedelta object when the rules are loaded.

self.matches: This is where ElastAlert 2 checks for matches from the rule. Whatever information is relevant to the match (generally coming from the fields in Elasticsearch) should be put into a dictionary object and added to self.matches. ElastAlert 2 will pop items out periodically and send alerts based on these objects. It is recommended that you use self.add_match(match) to add matches. In addition to appending to self.matches, self.add_match will convert the datetime @timestamp back into an ISO8601 timestamp.

self.required_options: This is a set of options that must exist in the configuration file. ElastAlert 2 will ensure that all of these fields exist before trying to instantiate a RuleType instance.

add_data(self, data):

When ElastAlert 2 queries Elasticsearch, it will pass all of the hits to the rule type by calling add_data. data is a list of dictionary objects which contain all of the fields in include, query_key and compare_key if they exist, and @timestamp as a datetime object. They will always come in chronological order sorted by ‘@timestamp’.

get_match_str(self, match):

Alerts will call this function to get a human readable string about a match for an alert. Match will be the same object that was added to self.matches, and rules the same as self.rules. The RuleType base implementation will return an empty string. Note that by default, the alert text will already contain the key-value pairs from the match. This should return a string that gives some information about the match in the context of this specific RuleType.

garbage_collect(self, timestamp):

This will be called after ElastAlert 2 has run over a time period ending in timestamp and should be used to clear any state that may be obsolete as of timestamp. timestamp is a datetime object.

Tutorial

As an example, we are going to create a rule type for detecting suspicious logins. Let’s imagine the data we are querying is login events that contains IP address, username and a timestamp. Our configuration will take a list of usernames and a time range and alert if a login occurs in the time range. First, let’s create a modules folder in the base ElastAlert 2 folder:

$ mkdir elastalert_modules
$ cd elastalert_modules
$ touch __init__.py

Now, in a file named my_rules.py, add

import dateutil.parser

from elastalert.ruletypes import RuleType

# elastalert.util includes useful utility functions
# such as converting from timestamp to datetime obj
from elastalert.util import ts_to_dt

class AwesomeRule(RuleType):

    # By setting required_options to a set of strings
    # You can ensure that the rule config file specifies all
    # of the options. Otherwise, ElastAlert 2 will throw an exception
    # when trying to load the rule.
    required_options = set(['time_start', 'time_end', 'usernames'])

    # add_data will be called each time Elasticsearch is queried.
    # data is a list of documents from Elasticsearch, sorted by timestamp,
    # including all the fields that the config specifies with "include"
    def add_data(self, data):
        for document in data:

            # To access config options, use self.rules
            if document['username'] in self.rules['usernames']:

                # Convert the timestamp to a time object
                login_time = document['@timestamp'].time()

                # Convert time_start and time_end to time objects
                time_start = dateutil.parser.parse(self.rules['time_start']).time()
                time_end = dateutil.parser.parse(self.rules['time_end']).time()

                # If the time falls between start and end
                if login_time > time_start and login_time < time_end:

                    # To add a match, use self.add_match
                    self.add_match(document)

    # The results of get_match_str will appear in the alert text
    def get_match_str(self, match):
        return "%s logged in between %s and %s" % (match['username'],
                                                   self.rules['time_start'],
                                                   self.rules['time_end'])

    # garbage_collect is called indicating that ElastAlert 2 has already been run up to timestamp
    # It is useful for knowing that there were no query results from Elasticsearch because
    # add_data will not be called with an empty list
    def garbage_collect(self, timestamp):
        pass

In the rule configuration file, examples/rules/example_login_rule.yaml, we are going to specify this rule by writing

name: "Example login rule"
es_host: elasticsearch.example.com
es_port: 14900
type: "elastalert_modules.my_rules.AwesomeRule"
# Alert if admin, userXYZ or foobaz log in between 8 PM and midnight
time_start: "20:00"
time_end: "24:00"
usernames:
- "admin"
- "userXYZ"
- "foobaz"
# We require the username field from documents
include:
- "username"
alert:
- debug

ElastAlert 2 will attempt to import the rule with from elastalert_modules.my_rules import AwesomeRule. This means that the folder must be in a location where it can be imported as a Python module.

An alert from this rule will look something like:

Example login rule

userXYZ logged in between 20:00 and 24:00

@timestamp: 2015-03-02T22:23:24Z
username: userXYZ