DevOps

T02 Deep Dive: Harvesting the Web Automatically: Screen Scraping, SOAP, XML and REST with PowerShell

August 8th, 2017

9:45am - 12:45pm

Level: Introductory Intermediate

Mark Minasi

IT Consultant, Author, Speaker

MR&D

The day stars the same for many IT pros — surfing to catch up. We skim our Twitter feeds, check out The Picture of the Day from some amazing photo site, check the weather, peek at eBay to see if that prized item is selling under $100 yet, or whatever. That's great, but what isn't so great is the fact that most of us are gathering those data the same way we've been doing it for it for twenty years — by clicking around in a web browser. It's the 21st century, and IT pros hoping to remain employed are automating things. It's high time to turn The Robot Hand of PowerShell to our daily web data-gathering tasks, whether personal- or business-related.

PowerShell includes a number of essential web-harvesting cmdlets (including Invoke-WebRequest and Invoke-Restmethod, amongst others). These help you automate web surfing and data interpretation, but unless you're familiar with the workings of "screen scraping," "web services," "SOAP" and the like, you probably won't have time to figure them out. So come along to this session by best-selling tech author and speaker Mark Minasi, "the Chief Explainer of the Really Complex." This session assumes no knowledge of PowerShell, SOAP, XML, REST or JSON. You'll be introduced to them all on the way. Attend this session and become the Master of Web Automation.

You will learn:

  • How Invoke-WebRequest and some regular expressions can let you "screen scrape," plucking just the one important bit of data from a web page automatically
  • About the basics of SOAP and REST-type web services