One of my friend was seeking my help creating a script to download bulk files and folder from internal office training web portal, just newly created. Folders and files web structure was looking like below. The first url link shows what are the different training material was available, downloading one file at time was going to take long time. Once I click on any of the required link, It has the PPTs, video files and folders stored into, further directories.
To work this script I need the link URL on the second browser window, Once ps1 script is executed, It asks for main two Parameters, First is Downloadurl and second is DownloadToFolder. Once url is validated it will start downloading the file and shows nice tree view, If any of the file is not downloadable or errors into 404, it will give me message in red.
All the info and main cmdlet of the script is Invoke-WebRequest, Which fetch information from web site. Once script is execution is complete, all files are downloaded, you can view the download folder, I further drilled down folders and viewed, files they are there.
Download this script here, it is also available on github.com.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
<# #requires -version 3 .SYNOPSIS Downlaod all files and folder from IIS directory website. .DESCRIPTION The Start-DirDownload cmdlet downloads complete directory and files from web. .PARAMETER Downloadurl Prompts you for download url .PARAMETER DownloadToFolder Prompts where you want to download files and folder from IIS web, DownloadPath is alias .INPUTS No Input .OUTPUTS Output is on console directly. .NOTES Version: 2.0 Author: Kunal Udapi Creation Date: 12 February 2017 Purpose/Change: Download automated way to download files from net (http://kunaludapi.blogspot.in) Useful URLs: http://vcloud-lab.com .EXAMPLE 1 PS C:\>Start-DirDownload -Downloadurl http://freetrainings001.com/trainingPortal/AzureAdvanced -DownloadToFolder C:\Temp This command start download files from given url and downloads to given folderpath. #> [CmdletBinding(SupportsShouldProcess=$True, ConfirmImpact='Medium', HelpURI='http://vcloud-lab.com')] Param ( [parameter(Position=0, Mandatory=$true, ValueFromPipelineByPropertyName=$true)] [String]$Downloadurl = 'http://freetrainings001.com/Office%20Learning%20Portal/Powershell%20Advanced%20Training/', [parameter(Position=1, Mandatory=$true,ValueFromPipelineByPropertyName=$true)] [alias('DownloadPath')] [String]$DownloadToFolder = 'C:\Temp\test' ) process { try { if (!(Test-Path -Path $DownloadToFolder)) { New-Item -Path $DownloadToFolder -Type Directory -Force -ErrorAction Stop | Out-Null } $CMDBrowser = Invoke-WebRequest $downloadurl -ErrorAction Stop $AllLinks = $CMDBrowser.links | Where-Object {$_.innerHTML -ne "[To Parent Directory]" -and $_.innerHTML -ne 'web.config'} #| Select -Skip 23 foreach ($link in $AllLinks) { $FolderName = $link.innerText $DownloadPath = Join-Path $DownloadToFolder $FolderName if (!(Test-Path $DownloadPath)) { New-Item -Path $DownloadPath -ItemType Directory | Out-Null } $RawWebsite = $Downloadurl -split '/' $WebSite = $RawWebsite[0,2] -join '//' Write-Host $FolderName -BackgroundColor DarkGreen $FolderUrl = "{0}{1}" -f $WebSite, $link.href #$Downloadurl Replacedby $WebSite $FolderLinks = Invoke-WebRequest $FolderUrl if ($FolderLinks.StatusCode -eq 200) { $FilesLinks = $FolderLinks.Links | Where-Object {$_.innerHTML -ne '[To Parent Directory]' -and $_.innerHTML -ne 'web.config'} #| Select-Object -Skip 5 foreach ($File in $FilesLinks) { $FileUrl = "{0}{1}" -f $WebSite, $File.href $FilePath = "{0}\{1}" -f $DownloadPath, $File.innerText try { if (!(Test-Path -Path $FilePath)) { Invoke-WebRequest -Uri $FileUrl -OutFile $FilePath Write-Host "`t|---$($File.innerText)" } else { Write-Host "`t|---$($File.innerText) --Already exist" -ForegroundColor DarkYellow } } catch { Write-Host "`t|---$FilePath --skipped" -BackgroundColor DarkRed } } } } } catch { Write-Host $error[0] } } |
Useful Scripts
Different ways to bypass Powershell execution policy :.ps1 cannot be loaded because running scripts is disabled
INSTALLING .NET 3.5 FRAMEWORKS ON WINDOWS SERVER 2012 R2
CONVERTING FROM SERVER 2012 CORE INSTALL TO FULL GUI
Installing, importing and using any module in powershell
How to Install and Use Microsoft PowerShell on Linux
