How to scrape Yahoo Finance stock data with Python
This time, we are going to learn the hands-on ability to scrape Yahoo financial data. Set-up python environment. Yahoo Finance page. For example, Alphabet Inc. (GOOG). Scrape & parse the page. The page we will be scrape. In the “network” page, we can’t find the json data. You can get all the data from the page source in script. View page source - script - root.App.main from bs4 import BeautifulSoup import re import json import requests response = requests.get("https://finance.yahoo.com/quote/GOOG?p=GOOG&.tsrc=fin-srch") soup = BeautifulSoup(response.text, "html.parser") ## print(soup.prettify()) script = soup.find('script', text=re.compile('root.App.main')).text data = json.loads(re.search("root.App.main\\s+=\\s+({.*})", script).group(1)) stores = data["context"]["dispatcher"]["stores"] print(stores) Response data ...
