Selenium is a project focused on automating web browsers. You need to follow the installation instructions for Docker Toolbox on Windows or Ubuntu. Docker is used to run applications by using containers. Containers are simply a bundle of libraries and other dependencies in one package. Scraping Dynamic Web Pages with C# and Selenium vainolo Programming 2020-10-12 3 Minutes A while back I wrote a post on how to scrape web pages using C# and HtmlAgilityPack (It was in May? Selenium is a tool designed to help you run automated tests in web applications. It is available in several different programming languages. Although it’s not its main purpose, Selenium is also used in Python for web scraping, because it can access JavaScript-rendered content (which regular scraping tools like BeautifulSoup can’t do).
A while back I wrote a post on how to scrape web pages using C# and HtmlAgilityPack (It was in May? So long ago? Wow!). This works fine for static pages, but most web pages are dynamic apps where elements appear and disappear when you interact with the page. So we need a better solution.
Selenium uses a web-driver package that can take control of the browser and mimic user-oriented actions to trigger desired events. This guide will explain the process of building a web scraping program that will scrape data and download files from Google Shopping Insights.
Selenium is an open-source application/library that let’s us automate browsing using code. And it is awesome. In this tutorial, I’m going to show how to scrape a simple dynamic web page that changes when an element is clicked. A pre-requisite for this tutorial is having the Chrome browser installed in your computer (more on that later).
Let’s start by creating a new .NET core project:
To use Selenium we need two things: a Selenium WebDriver which interacts with the browser, and the Selenium library which connects our code with the Selenium WebDriver. You can read more in the docs. Gladly, both of them come as NuGet packages that we can add to the solution. We’ll also add a library that provides some Selenium extensions:
One important thing to note when you install these packages is the version of the Selenium.WebDriver.ChromeDriver that is installed, which looks something like this: PackageReference for package 'Selenium.WebDriver.ChromeDriver' version '85.0.4183.8700'. The major version of the driver (in this case 85) must match the major version of the Chrome browser that is installed on your computer (you can check the version you have by going to Help->About in your browser).
To demonstrate the dynamic scraping, I’ve created a web page that has the word “Hello” on it, that when clicked, adds the word “World” below it. I’m not a web developer and don’t pretend to be, so the code here is probably ugly, but it does the job:
I added this page to the project and defined that the page must be copied to the output directory of the project so it can be easily consumed by the scraping code. This is achieved by adding the following lines to the DynamicWebScraping.csproj project file somewhere between the opening and closing Project nodes:
Python Scrapy Selenium
The scraping code will navigate to this page and wait for the heading1 element to appear. When it does it will click on the element and wait for the heading2 element to appear, fetching the textContent that is located in that element:
Let’s build and run the project:
The program opens a browser window and starts to interact with it, returning the text inside the second heading. Pretty cool, right? I have to admit that the first time I running this it feels really powerful, and opens a whole new world of things to build… If only I had more time :-).
As always, the full source code for this tutorial can be found in my GitHub repository.
Web Scraping Open Source
Selenium Web Scraping Java
Hoping that the next post comes sooner. Until next time, happy coding!