Scraping Champlain Classes

Final Scripts:

Functions:

SYS-320-Automating-and-Scripting/week4_5/WebScraping.ps1 at main · Hsanokklis/SYS-320-Automating-and-ScriptingGitHub

Function Calls and Queries:

SYS-320-Automating-and-Scripting/week4_5/Call_WebScraping.ps1 at main · Hsanokklis/SYS-320-Automating-and-ScriptingGitHub

Put the following html file in your "C:\xampp\htdocs" directory. Make sure your web server is on and you can access to your page from a web browser.

604KB

Courses.html

Open

Unsolved Script to Scrape Classes

Solved Script to Scrape Classes

Time End is omitted from the script

function gatherClasses(){

$page = Invoke-WebRequest -TimeoutSec 2 http://10.0.17.37/courses.html

#Get all the tr elements of the HTML page 
$trs= $page.ParsedHtml.body.getElementsByTagName("tr")

#Empty array to hold results 
$FullTable = @()
for($i=1; $i -lt $trs.length; $i++){ #going over every tr element
    
    #Get every td element of each tr element 
    $tds = $trs[$i].getElementsByTagName("td")
    
    # Want to seperate start time and end time into different fields
    $Times = $tds[5].innerText.Split("-")

    $FullTable += [pscustomobject]@{"Class Code" = $tds[0].innertext;
                                    "Title"      = $tds[1].innertext;
                                    "Days"       = $tds[4].innertext;
                                    "Time Start" = $Times[0];
                                    "Instructor" = $tds[6].innertext;
                                    "Location"   = $tds[9].innertext;
                                    }
} #end of for loop
return $FullTable
}

# call function to test that it works! 
gatherClasses

Script Working

Call Script with Dot Notation

# Dot Notation 
. C:\Users\champuser\SYS-320-Automating-and-Scripting\week4_5\WebScraping.ps1

# Call function 
gatherClasses

Turn Days Property into an Array

The Days property displays its output as letters such as M, T, W, Th, and F. We want to change this so that we are outputting the days as their full names. We can turn the days property into an array with a translator function. Write a translator function where your first function is.

Unsolved Days Array Table

Solved Days Array Table

#Funcation to turn days property into an array

function daysTranslator($FullTable){

# Go over every record in the table 
for($i=0; $i -lt $FullTable.length; $i++){

    #Empty array to hold days for every record
    $Days = @()

    #If you see "M" --> Monday 
    if($FullTable[$i].Days -ilike "M") {$Days += "Monday"}

    #If you see "T" followed by T,W or F --> Tuesday 
    if($FullTable[$i].Days -ilike "T[W,F]") {$Days += "Tuesday"}

    #If you see "W" --> Wednesday 
    if($FullTable[$i].Days -ilike "W") {$Days += "Wednesday"}

    #If you see "TH" --> Thursday  
    if($FullTable[$i].Days -ilike "TH") {$Days += "Thursday"}

    #If you see "F" --> Friday 
    if($FullTable[$i].Days -ilike "*F") {$Days += "Friday"}

    #Switch from abbreviation to full name
    $FullTable[$i].Days = $Days

} #end of for loop
return $FullTable
}

Call Both Scripts with Dot Notation

# Dot Notation 
. C:\Users\champuser\SYS-320-Automating-and-Scripting\week4_5\WebScraping.ps1


# Assigning the gatherClasses function to the variable $classes   
$classes = gatherClasses

#Assigning the daysTranslator function to the variable $DaysTranslator to call it with the $classes variable
$DaysTranslator = daysTranslator $classes

Making specific Queries

List all Classes taught by Furkan Paligu

Unsolved:

Solved:

# List all the classes with the Instructor Furkan Paligu

$DaysTranslator | select "Class Code", Instructor, Location, Days, "Time Start"| `
where {$_."Instructor" -ilike "Furkan Paligu"}

List all the classes of JOYC 310 on Mondays, only display Class Code and Times. Sort by Start Time

Unsolved:

Solved:

#List all classes of JOYC 310 on Mondays, only display Class Code and Time. Sort by Start Time. 
$DaysTranslator | Where-Object {($.Location -ilike "JOYC 310") -and ($.days -contains "Monday")} |`
Sort-Object "Time Start" | `
Select-Object "Time Start", "Class Code"

Make a list of all the instructors that teach at least 1 course in one of the courses: SYS, NET, SEC, FOR, CSI, DAT. Sort by name and make it unique

Unsolved:

Solved:

$Instructors = $DaysTranslator | Where-Object { ($_."Class Code" -ilike "SYS*") -or `
                                                ($_."Class Code" -ilike "NET*") -or `
                                                ($_."Class Code" -ilike "FOR*") -or `
                                                ($_."Class Code" -ilike "CSI*") -or `
                                                ($_."Class Code" -ilike "DAT*")`
                                                }`
                                                 | Sort-Object "Instructor"`
                                                 | Select-Object "Instructor" -Unique 
#variable call 
$Instructors

Group all the instructors by the number of classes they are teaching, sort by the number of classes they are teaching

Unsolved:

Solved:

# 1. Group all instructors by the number of classes they are teaching 
# 2. sort by the number of classes they are teaching

$DaysTranslator | Where-Object { $_.Instructor -in $Instructors.Instructor}`
                | Group-Object "Instructor"| Select-Object Count, Name | Sort-Object Count -Descending