Web Scraping with Selenium Web Driver
In this blog post, I would discuss how to successfully web scrap the TripAdvisor website.
Tripadvisor is a web database that archives traveler's experiences at various locations. From attraction to various cuisines, you have access to an abundance of options at your fingertips.
WebDriver drives a browser natively, as a user would, either locally or on a remote machine using the Selenium server, marks a leap forward in terms of browser automation. This tool was used to web scrap the different attractions and different destinations. I chose this route because if I wanted you to use the traditional Beautiful Soup web scraping technique, I would have to generate all the different URLs and it would have been time costly.
Here is some example code:
So with this section of code, you will be able to open the webpage and inpur the destination.
The page was dynamically changing so we had to write in a portion to handle pop-ups. Notice there are a lot of try-except statements because we don't want the driver to give an error if the button is not present.
Once we get to the location attraction page, we then want to store all the URLs into a list and iterate through and grab all the important information and save it to a CSV file.
