How To Install Python Scrapy Correctly

There is no difference between installing Scrapy and installing other Python packages, just to run the pip install scrapy command to install it. If you run the command in a command-line window, you will see that the program does not download and install python Scrapy immediately, but a large number of third-party packages are constantly being downloaded and installed first. This is because python Scrapy requires a large number of third-party packages. Typically, Scrapy requires below third-party packages.

1. Scrapy Required Python Libraries.

1. cryptography: Python library for encryption.
2. cssselect: Python extension package for handling CSS selectors.
3. CFFI: Python’s interface for calling C programming language.
4. lxml: A library for processing XML and HTML documents, better than Python’s built-in XML module.
5. pyOpenSSL: Python package for supporting SSL (security socket layer).
6. Twisted: Event-based network engine package for Python.
7. zope.interface: Provides an extended library for Python’s lack of interfaces.

If there are no such third-party packages in your Python environment, python will automatically download and install them according to the requirements. PIP will automatically complete the whole process. We just need to wait.

2. Install Scrapy Required Python Library Twisted.

1. However, pip will prompt the following error when downloading and installing the Twisted python library automatically.

   error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

2. The above error is only because the Twisted installation package downloaded by pip has some defects, so you can download the Twisted installation package by yourself follow the below steps.
3. Go to https://pypi.org/project/Twisted/#files to find related Twisted installation package.
4. Twisted provides corresponding installation packages for different Python versions, such as Python 2.7, 3.4, 3.5, 3.6, 3.7.
5. If your Python version is python3.6, you should download the Twisted installation package for python 3.6 version, the file name contains cp36 ( Twisted-20.3.0-cp36-cp36m–win_amd64.whl ).
6. The file name with win2 is the 32-bit version, and the file name with win_amd64 is the 64-bit version. You need to select the corresponding version according to the OS bit version.
7. Run the following command to install the Python Twisted package.

   $ pip install Twisted-20.3.0-cp36-cp36m-macosx_10_6_intel.whl

8. After the installation is complete, it will prompt the following installation success message.

   Successfully installed Scrapy-20.3.0

9. Execute the pip install scrapy command again to install the Scrapy until successful.
10. Now run command $ python -m pydoc -p 8899 in a terminal, it will start an embedded web server.

    $ python -m pydoc -p 8899
    Server ready at http://localhost:8899/
    Server commands: [b]rowser, [q]uit
    server>

11. Press character b in the server console interface or open url http://localhost:8899/ in a web browser, you can see the python Scrapy document in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages area.

    /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
          	 	
    OpenSSL (package)
    ......
    scrapy (package)
    ......