Tags

, ,


I’ve not posted for ages but here finally is something new.  If you stream on Twitch or moderate any channels, you can use the following script to make a database of viewers and how often they visit your channel.  The requirement is that you’ve been in the channel’s chat via mIRC during the stream with time-stamped logging enabled.  The script parses the log and makes/updates a database of usernames, first visit date, last visit date and total visits.  Since the database is a human-parsable textfile with an ordered list of users one per line, there’s also a comments field in case you want to manually note something like “Troll”, “Donator” or “FailFish” against anyone.
OK, I admit it’s probably not of much practical use but I did it mainly for the challenge once I’d got thinking about it! Maybe you’ll find some interesting ideas – or laugh at where I’ve made things harder for myself than I needed to!

Firstly, here’s the code. Sorry I had to take all the single-quoted strings and convert them to double-quoted to stop the WordPress colouring from breaking:

Function Parse-mIRC {
    #requires -Version 3
    Param(
        [Parameter(Mandatory=$True)]
        [string]$databasePath,
        [Parameter(Mandatory=$True)]
        [string]$channel
    )

#region Init
    $logRootPath = "$($env:APPDATA)\mIRC\Logs"
    #eg [11:41] * nightbot (nightbot@nightbot.tmi.twitch.tv) has joined #monstercat'
    $regex = "^\[[0-9][0-9]:[0-9][0-9]\] \* (?<Username>[A-Za-z0-9_\-]+) \(" # User joins or parts channel
    $imported = 0
    $added = 0
    $updated = 0
    $logs = 0
#endregion

#region Create/import database
    if ((Test-Path -Path $databasePath) -eq $False) {
        Write-Verbose "*** Creating new database $databasePath ***"
        New-Item -Path $databasePath -ItemType File -Value '' | Out-Null
    } else {
        Write-Verbose "*** Importing database ***"
    }
    $fromJson = Get-Content -Path $databasePath | ConvertFrom-Json
    $viewers = @{}
    if ($fromJson -ne $null) {
        $fromJson | ForEach-Object {
            $viewers.Add($_.Username, $_)
            Write-Verbose ("Importing user '{0}'..." -f $_.Username)
            $imported += 1
        }
    }
    Write-Verbose ("Imported {0} users" -f $imported)
#endregion

#region Do Parse
    [Array]$logFiles = Get-ChildItem -Path "$logRootPath\#$($channel).*.log"
    if ($logFiles.Count -gt 0) {
        foreach ($logFile in $logFiles) {
            $fileName = $logFile.Name
            # Need to use date log started, not date when script run in case run after midnight.
            $streamDate = $logFile.CreationTime.ToShortDateString()
            $logContents = Get-Content -Path $logFile
            Write-Verbose ("*** Parsing logfile '{0}' ***" -f $fileName)
            foreach ($line in $logContents) {
                if ($line -match $regex) {
                    $username = $matches.Username
                    if ($viewers.ContainsKey($username) -eq $False) {
                        Write-Verbose ("Adding new user '{0}'..." -f $username)
                        $props = @{
                            Username = $username;
                            FirstSeen = $streamDate;
                            LastSeen = $streamDate;
                            StreamCount = 1;
                            Comments = ''
                        }
                        $newViewer = New-Object -TypeName PSObject -Property $props
                        $viewers.Add($username, $newViewer)
                        $added += 1
                    } else {
                        if ($viewers[$username].LastSeen -ne $streamDate) {
                            Write-Verbose ("Updating user '{0}'..." -f $username)
                            $viewers[$username].LastSeen = $streamDate
                            $viewers[$username].StreamCount = $viewers[$username].StreamCount + 1
                            $updated += 1
                        }
                    } #contains key
                } #specific line
            } #logFile lines
        } #logPaths
        $logs += 1
    }
#endregion

#region Write updated database
    Write-Verbose "*** Updating database ***"
    $viewers.Keys | Sort-Object | ForEach-Object {
        $viewers[$_] | ConvertTo-Json -Compress
    } | Set-Content -Path $databasePath -Force
    Write-Verbose ("Added {0} and updated {1} viewers from {2} log files." -f $added, $updated, $logs)
#endregion
}

Parse-mIRC -databasePath "C:\Users\Me\Documents\mIRC-Monstercat.db" -channel "monstercat" -Verbose

Having come across JSON when playing with the Twitch API previously, I was looking into the ConvertTo-Json Cmdlet as a way of making PSObjects storable in some way when I discovered it has a -Compress parameter which makes the JSON object a string with no whitespace, indents or newlines. This became the solution to create a simple database – a text file where each line is a JSON object representing a stream viewer.
Here’s an example line taken from the database this script creates:

{"Username":"monstercat","FirstSeen":"11/04/2015","LastSeen":"12/04/2015","StreamCount":2,"Comments":"Music!"}

Initially, the existing database file is read in by Get-Content and piped to ConvertFrom-Json which spits out a bunch of PSObjects – one for each viewer which is stored in a collection (line 27).

$fromJson = Get-Content -Path $databasePath | ConvertFrom-Json

Given a collection of such objects representing past viewers, I wanted a way to be able to find if a given username was already listed or to update an existing entry. This made me think of dictionaries and looking for a given Key. I don’t know if this is the best solution but it’s what I did. I create an empty hashtable and then go through the collection of PSObjects and add each to the hashtable using the Add() method of the hashtable (line 31). For each addition, the Key is taken from the Username property of the PSObject and the Item is the entire PSObject itself.

$fromJson | ForEach-Object { $viewers.Add($_.Username, $_) }

The mIRC logfiles for the channel in question are obtained from the relevant %AppData% folder and parsed in turn. Twitch reveals the Parts and Joins in the log which you don’t see through their web interface. Here’s an example:

[11:41] * nightbot (nightbot@nightbot.tmi.twitch.tv) has joined #monstercat

By parsing the log, the usernames are matched and extracted with a Regular Expression (line 13). I’m sure the one I used can be simplified!

$regex = "^\[[0-9][0-9]:[0-9][0-9]\] \* (?<Username>[A-Za-z0-9_\-]+) \("

Note the named grouping which is used to pull the Username out of a match (line 50):

if ($line -match $regex) { $username = $matches.Username }

The hashtable’s ContainsKey() method is used to see whether or not this user has been seen before.

if ($viewers.ContainsKey($username) -eq $False) { # etc

If the username already exists in the hashtable, the LastSeen date is compared to the stream’s date to see if the entry has been updated yet whilst parsing this logfile. If not, the existing entry has its properties updated (lines 66 and 67). The relevant PSObject for the user and its properties are accessed by indexing into the hashtable using the $hashtable[“KeyName”] syntax. Note also how I got the date the log file was written into the right format.

$streamDate = $logFile.CreationTime.ToShortDateString()
if ($viewers[$username].LastSeen -ne $streamDate) { # etc

If the user is a new one, a new PSObject is created to add into the hashtable (lines 53 to 61).

$props = @{
    Username = $username;
    FirstSeen = $streamDate;
    LastSeen = $streamDate;
    StreamCount = 1;
    Comments = ''
}
$newViewer = New-Object -TypeName PSObject -Property $props
$viewers.Add($username, $newViewer)

Once the entire log file has been parsed, I wanted to write it back to disc such that the usernames were in order. The way I did this was to get the list of keys of the hashtable (ie the usernames) via the Keys property, sort on them and then pass them on to a ForEach-Object. The latter’s script block uses each key to index back into the hashtable, pull out the PSObject and turn it back into a line of JSON (line 81) with the whole lot sent on to Set-Content:

$viewers.Keys | Sort-Object | ForEach-Object {
    $viewers[$_] | ConvertTo-Json -Compress
} | Set-Content -Path $databasePath -Force

This script assumes the streamer has streamed only once per day since I only use a date stamp with no time. Also, a new log file is created at midnight by mIRC so unless you concatenate the logs of a stream that goes past midnight, you will mess up your stats and record double viewings for a single stream.

Advertisements