In order to follow this lesson, you will need to make sure the following software is installed on your computer.
For the first half of the lesson, we will use a Chrome browser extension to get started with web scraping.
The second part of the lesson requires the Python programming language and access to a command-line interface (shell) on your computer.
Please note: for the NYU Libraries Library Carpentry workshop, you can access everything you need for this lesson via a web browser by using NYU’s JupyterHub
If you would prefer to work locally on your own machine, please refer to the Software Carpentry setup instructions for the Bash shell and Python if you need guidance.
Prerequisites
This part of the lesson requires some prior knowledge of Python and how to use a shell. If you need help getting started on those topics, we suggest going through the following lessons first (during a workshop or on your own):
Once you have a working installation of Python, the next step is to install Scrapy.
If you have installed Python using the Anaconda framework as suggested by the Software Carpentry setup instructions, you can easilly install Scrapy by doing the following:
conda install -c conda-forge scrapy
Alternatively, if you have another distribution of Python, you can try using pip:
pip install Scrapy
If you run into issues while installing Scrapy, refer to the official Scrapy install guide or get in touch with your lesson instructor.