Script to enumerate Windows events with name, ID, security monitoring recommendation, URL

When setting up and tuning a SIEM solution, you will write a lot of rules to detect well-known and arising security threats. If you need to protect systems running on Windows, you can use Microsoft’s security recommendations for each event. On its knowledgebase, Microsoft lists many hundred events with their ID and security monitoring recommendations. For example, if you have a look at the event with ID 1102: “Audit log was cleared”, the security recommendation reads:

Typically you should not see this event. There is no need to manually clear the Security event log in most cases. We recommend monitoring this event and investigating why this action was performed.

So, we should certainly implement a rule in our SIEM that looks for such audit log cleared events. Now, we can of course look up the recommendations for each event, but not all event IDs are described and not all events do have security recommendations. Some websites list different event IDs and also give recommendations on how to monitor them, but recommendations are not identical to the ones from Microsoft and the list of event IDs is also usually not 100% complete.

Therefore, I wrote a Python script that enumerates all Microsoft Windows Event IDs. If Microsoft has something to say that is related to security (i.e. a webpage is published), the script will parse specifically the monitoring recommendations part (which is there even if no recommendation is given) using regular expressions. Furthermore, the script will try to understand if there is a meaningful monitoring recommendation in the corresponding part, or if it just says something that reads along the lines of “no security monitoring recommendations”.

For each event, the script will store the HTML part of the security recommendation, and also a Boolean value on if it thinks that Microsoft has a security monitoring recommendation for it or not. All found events will then be saved as an Excel table that contains event name, ID, URL, the HTML part for monitoring recommendations, as well as the flags “has recommendation” and “applicable for result=success” or “result=failure”.

With this table you can quickly sort for these events where “has monitoring recommendation” is true, have a look at the recommendations and implement these in your SIEM solution. Ideally, you write your monitoring rules in SIEM-agnostic Sigma notation. With this you will be able to use the same rules for different SIEM systems, for example QRadar, Splunk, or sumo logic.

You can run the script with the code below. Or you can simply use the list which I already created with it:

__author__ = "Fabian Voith"
__version__ = "1.0.0"
__email__ = "admin@fabian-voith.de"

import pandas as pd
import requests
import re
import warnings
from tqdm import tqdm

warnings.filterwarnings("ignore")


class pageManager:
    # static class, no instance, to organize event pages
    __pages = set()
    
    def enumeratePages():
        for page in pageManager.__pages:
            print(page)
            
    def countPages():
        return(len(pageManager.__pages))

    def addPage(page):
        pageManager.__pages.add(page)
        
    def purge():
        pageManager.__pages.clear()
        
    def saveToExcel(fileName):
        titles = []
        eventids = []
        flags = []
        links = []
        hasRecommendation = []
        recommendations = []
        
        colEVENTNAME = 'Event Name'
        colEVENTID = 'Event ID'
        colRESULT = 'Result'
        colHASRECOMMENDATION = 'Has Recommendation'
        colRECOMMENDATIONTEXT = 'Security Monitoring Recommendation'
        colURL = 'URL'
        
        for page in pageManager.__pages:
            titles.append(page.getTitle())
            eventids.append(page.getEventID())
            flags.append(page.getResultFlag())
            hasRecommendation.append(page.hasRecommendation())
            recommendations.append(page.getRecommendationText())
            links.append(page.getLink())
            
        table = {colEVENTNAME: titles, colEVENTID: eventids, colRESULT: flags, colHASRECOMMENDATION: hasRecommendation, colRECOMMENDATIONTEXT: recommendations, colURL: links}

        df = pd.DataFrame(table, columns = [colEVENTNAME, colEVENTID, colRESULT, colHASRECOMMENDATION, colRECOMMENDATIONTEXT, colURL])
        df[colEVENTID] = pd.to_numeric(df[colEVENTID])
        df.sort_values(by=[colEVENTID], inplace=True)
        df.to_excel(fileName, index=False)
        
 

class eventPage:
    # represents each individual page that we found
    
    def __init__(self, link, title, resultFlag, eventid, recommendationText, hasRecommendation):
        self.__link = link
        self.__title = title
        self.__resultFlag = resultFlag
        self.__eventid = eventid
        self.__recommendationText = recommendationText
        self.__hasRecommendation = hasRecommendation
        
    def getTitle(self):
        return(self.__title)
    
    def getLink(self):
        return(self.__link)
    
    def getResultFlag(self):
        return(self.__resultFlag)
    
    def getEventID(self):
        return(self.__eventid)
    
    def getRecommendationText(self):
        return(self.__recommendationText)
    
    def hasRecommendation(self):
        return(self.__hasRecommendation)
    
    def __repr__(self):
        return(f'Event "{self.getTitle()}" with ID {self.getEventID()} for result {self.getResultFlag()} is on {self.getLink()}')
    

class pageCreator:
    # static class to help us create proper pages that we later can save
    
    # prepare regular expressions for title...
    reUncleanTitle = re.compile(r'<h1.*>(.*)</h1>') # Title with all flags and event ID
    reCleanTitle = re.compile(r'.+: (.*)')          # Only title, without flags and ID
    reFlag = re.compile(r'\((.+)\):')        # Success, Failure, Both
    reEventID = re.compile(r'\d+')           # Numeric ID of the event
    # ... and for recommendation:
    # Even if an event does not have security recommendations, it still has the related section
    reRecommendationPart = re.compile(r'Security Monitoring Recommendations</h2>(.*)<!-- </content> -->', re.DOTALL)
    # If the part for recommendations contains something like "no recommendation" or
    # "no additional recommendations" we assume that no recommendation is given
    reHasNoRecommendationHint = re.compile(r'no\s\w*?\s?recommendation')
    
    def exists(path):
        # checks if website exists
        # since we are simply enumerating many event IDs, we will find many non-existant pages
        r = requests.head(path)
        return r.status_code == requests.codes.ok
    
    def create(link):
        # create page from scratch and do all necessary checks
        page = None
        if exists(link):
            html = requests.get(link, verify=False)
            
            # we need to fix the encoding, otherwise some characters will look very awakward
            html.encoding = html.apparent_encoding
            html = html.text
            
            title, flag, eventid = pageCreator.__parseTitle(html)
            
            recommendation, hasRecommendation = pageCreator.__parseRecommendation(html)
            
            page = eventPage(link, title, flag, eventid, recommendation, hasRecommendation)
            
        return(page)
    
    def __parseTitle(html):
        # parse title of web page to get some important information about event
        
        # with this, the title still contains (S, F) for Failure, Success, e.g. 4656(S, F): A handle to an object was requested.
        unclean_title = pageCreator.reUncleanTitle.search(html).group(1)
        
        # get result flag, for which event description is valid (S = Success, F = Failure, - = None)
        flag = pageCreator.reFlag.search(unclean_title).group(1)
        
        # get event id:
        eventid = pageCreator.reEventID.search(unclean_title).group(0)

        # remove result flag from title:
        clean_title = pageCreator.reCleanTitle.search(unclean_title).group(1)
        
        return(clean_title, flag, eventid)
    
    def __parseRecommendation(html):
        
        # Even if it says "no recommendation", we still assume that it DOES have a recommendation
        # if the part for recommendations is longer than or equal to 700 characters. In such cases "no recommendation"
        # usually only is meant for a small part in the recommendation area
        assumeRecommendationThreshold = 700
        
        # most events do have the recommendations part, but some very few like
        # https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4671
        # do not have it
        try:
            recommendationPart = pageCreator.reRecommendationPart.search(html).group(1)
            hasRecommendation = True
        except:
            return('', False)
        
        if len(recommendationPart) < assumeRecommendationThreshold:
            # the recommendation part is quite short, let's check if we find a hint that no recommendation is given
            # if the text is longer than the threshold, we always assume that there is some recommendation
            noRecommendation = pageCreator.reHasNoRecommendationHint.search(recommendationPart)
            
            if noRecommendation:
                # if we did find  a string like "no recommendation" or "no additional recommendations" in a short
                # paragraph, we think that there is no recommendation
                hasRecommendation = False
                
        
        # for the sake of completeness, we still return the recommendationText, 
        # even if we think that no recommendation was given
        return(recommendationPart, hasRecommendation)
        

# if we run the script using Jupyter notebook, our previous runs will
# have been stored by Jupyter, so we purge them first
pageManager.purge()
# "brute-force" relevant ID range 1000-6424
# we could also multi-thread this to make it faster
# we are using the tqdm library to show a nice progress bar
for eventid in tqdm(range(1000, 6425)):
    link = f'https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-{eventid}'
    
    page = pageCreator.create(link)
    
    if page != None:
        pageManager.addPage(page)
        

#pageManager.enumeratePages()
filename = 'windows-security-events.xlsx'
pageManager.saveToExcel(filename)
print(f'Saved {pageManager.countPages()} events to file {filename}.')

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.