This article was published as a part of the Data Science Blogathon.
In India, a pandemic has drawn more private and govt. sectors to adopt work from home. This resulted in people having more time to research investing in stocks. Many start-ups and apps came into the picture for investing in stocks. A lot of people wanted to use different tools and techniques to analyze stocks.
If you have coding knowledge and want to analyze nifty 50 historical data and get insights, then this article will you out.
Once we have the proper historical data, then we can analyze trends and start investing in stocks.
Now thinking about how to get nifty 50 historical data?
There are a couple of websites which provide nifty historical data.
Please find websites below which provide historical data.
1. https://www1.nseindia.com/
2. https://finance.yahoo.com/
3. https://in.investing.com/
In this article, we will concentrate on investing.com. We will see how to extract Nifty 50 historical data from this website.
ll scrap nifty 50 data by passing the date range. After passing the date range, we will get data as below.
Web scraping tools or modules used for getting data from nifty data
Selenium is also used for web scraping. Selenium requires a webdriver to interface with the browser. Here we will use the Chrome browser. Selenium will use a chrome driver as a webdriver to control the chrome browser.
The request library or module is used for making HTTP requests to the site.
- BeautifulSoup
The BeautifulSoup library or module used for parsing HTML documents.
Below are the steps to be followed for the web scraping nifty historical data from investing.com
- Installing or downloading a chrome driver in Windows and Linux.
- Loading chrome driver using Selenium’s web driver for web scraping.
- Preparing Start Date and End Date for fetching data from investing.com.
- Constructing URL for extracting data from investing.com.
- Initializing selenium driver to get data.
- Getting the content of the page using Beautiful Soup.
- Extracting the required columns for Nifty data.
Installing or Downloading a Chrome Driver in Windows and Linux
For windows just download the required version of the chrome driver from the below-provided link and extract the chrome driver from the zip folder and store it in the same place where your python script is there
https://chromedriver.chromium.org/downloads
For Linux please follow the below steps
1. Run the below command to update ubuntu
apt-get update # to update ubuntu to correctly run apt install
2. Run the below command for installing the chrome driver
apt install chromium-chromedriver
3. Copy chrome driver to the bin folder
cp /usr/lib/chromium-browser/chromedriver /usr/bin
4. Add a path to the system path
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
5. Add below chrome options to webdriver
chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.add_argument('--disable-dev-shm-usage')
Selenium’s Web Driver for Web Scraping
For Windows use the below command to load selenium’s web driver
driver = webdriver.Chrome()
For Linux use the below command to load selenium’s web driver
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
Preparing Start Date and End Date for Fetching Data
Please find below the code for preparing the start date and end date. We will be pulling data from starting of the year to the current day.
The end date will be the current day and the start of the year will be the start date.
today = date.today() enddate = time.mktime(today.timetuple()) enddate = int(enddate) starting_day_of_current_year = datetime.now().date().replace(month=1, day=1) starting_day_of_current_year stdate=time.mktime(starting_day_of_current_year.timetuple()) stdate=int(stdate)
Constructing URL for Extracting Data from investing.com
Now we will construct a URL for fetching data from investing.com
url="https://in.investing.com/indices/s-p-cnx-nifty-historical-data?end_date={}&st_date={}".format(enddate,stdate)
We are passing the start date and end date to the URL.
Initializing Selenium Driver to get Data
Please find below the code which will invoke the selenium driver to get data from investing.com
The Below code will invoke the chrome browser by passing the above URL.
driver.get(url)
Getting the Content of the Page using Beautiful Soup
Please find below the code used to get the page content of the URL which we have passed with parameters.
#Get Page Content Data content = driver.page_source soup = BeautifulSoup(content)
Extracting Required Columns for Nifty 50 Data
Now we will prepare a list and extract the required columns as a list
We will initialize a list for each required column and search for the required column in the content that we got from the website.
#Loading Dates Column
date=[] for a in soup.findAll('td', attrs={'class':'col-rowDate'}): date_txt=a.find('span', attrs={'class':'text'}) date.append(date_txt.text)
The output of the above command will be as shown below
We will search for the required details and append them to the list.
We will perform similar activities in all required fields.
#Loading Closing Prices close=[] for a in soup.findAll('td', attrs={'class':'col-last_close'}): close_txt=a.find('span', attrs={'class':'text'}) close.append(close_txt.text) #Loading Open Prices open=[] for a in soup.findAll('td', attrs={'class':'col-last_open'}): open_txt=a.find('span', attrs={'class':'text'}) open.append(open_txt.text) #Loading High Prices high=[] for a in soup.findAll('td', attrs={'class':'col-last_max'}): high_txt=a.find('span', attrs={'class':'text'}) high.append(high_txt.text) #Loading Low Prices low=[] for a in soup.findAll('td', attrs={'class':'col-last_min'}): low_txt=a.find('span', attrs={'class':'text'}) low.append(low_txt.text)
Preparing DataFrame and Transforming Data
We will prepare a DataFrame with the required columns
## Prepare DataFrame df_nifty = pd.DataFrame({'Date':date,'Open':open,'High':high,'Low':l':close})
Below is the sample data after converting it into DataFrame.
If you see the above screenshot of data, the Date is not in proper format and the other columns’ data is in string.
Now we will format the data. First, remove commas in the column data and then convert them to the required format, like string to date and string to float.
df_nifty['Date'] = df_nifty['Date'].str.replace(r",","") df_nifty['Ddatetime(df_nifty.Date , format = '%b %d %Y') df_nifty=df_nifty.drop_duplicates(subset="Date") #droping dupicate data data = df_nifty data['Close']=data['Close'].str.replace(r",","").astype('float') data['Open']=data['Open'].str.replace(r",","").astype('float') data['High']=data['High'].str.replace(r",","").astype('float') data['Low']=data['Low'].str.replace(r",","").astype('float')
Sample data are shown below
Conclusion
To summarize, in this article we have learned how to extract nifty data from investing.com using web scraping in python. We got a basic understanding of installing Selenium in Windows and Linux. We also got a basic idea about finding an element from the content using BeautifulSoap and storing it in a list. We have also got a basic idea of combining all required columns into a DataFrame. We have also got a basic idea about cleaning the data and converting it into the proper format.
Now we have nifty historical daily data and we can start analyzing upward and downward trends.
We can roll up data to weekly, monthly and quarterly.
What can we do next?
- EDA of the extracted nifty data.
Start Exploratory Data Analysis of data by plotting different graphs , checking for missing data , checking for outliers , checking correlation etc.
- Convert the extracted data into time series.
Convert the extracted data into timeseries data using different libraries.
- Use multiple models for forecasting.
Once we convert data into time series, then we can use multiple models for forecasting, like Facebook’s NeuralProphet , AUTO ARIMA , Long Short Term Memory (LSTM) etc.
Please find below the GitHub link which has code for both Windows and Linux.
Fork the repo and implement the code accordingly.
https://github.com/bashamsc/nifty_web_scraping
The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.