11. Introduction to web-crawling in PythonΒΆ

This section contains an introduction to some basic python web crawling tools. We will focus on programs that request web services from service providers and programs that scrape data from web sites. Web service applications will involve us in a new kind of programming called client-server programming; the programs we will look at will be client programs making requests from service on the Internet. Although the underlying foundation of a web-scraping program is also a client-server interaction, we will use some tools that hide the details of those interactions, and allow us to fetch web page content directly. We will then look at how to extract patterned information from a web page.