Chris Conlan

Financial Data Scientist

Pulling All Sorts of Financial Data in Python [Updated for 2021]

By Leave a Comment

Contributed so kindly by Joe Wojniak

It may seem obvious, but financial research requires data — and a lot of it. If financial research isn’t your day job, it can be surprisingly difficult to come by. Here are some suggestions for acquiring data to use in your financial research project.

Data Sources

Every project has to start somewhere. In development, you need a data that is representative of the production data. This can be a historical or a static data set. In development, you keep a local copy and run your testing on it. In production, you update it and re-run your analysis on it regularly. Here are some great places to get this data.

Simulated End-of-day and Alternative Data

Financial research tends to produce unique efforts. Rarely does a researcher ask that their research be reproduced (and hence to be validated by being reproduced.) Chris Conlan has done something incredible here. Chris has provided Python trading algorithms and the data to reproduce the results. In a world where it seems that the zero-sum game rules, Chris has provided the methods and the data so that you can improve your trading chops. Check out the repo and accompanying book https://github.com/chrisconlan/algorithmic-trading-with-python.

Stooq for Multiple Asset Classes

See this site https://stooq.com/.

Stooq is a great website because it provides data for free. It’s a little too good to believe, but get your free data while it’s available for:

  • Indices
  • Currencies
  • Stocks
  • Bonds
  • Historical data – want to know what the Nasdaq was back in 1938? Stooq has it!. Just make sure you have plenty of bandwidth for this download. Daily, Hourly, and 5 minute data is available for all major exchanges. See this link https://stooq.com/db/h/

Yahoo Finance Still Exists

The Yahoo Finance API isn’t as easy to access as it used to be. But there is a work around. The Quantstat Github repo written by Ran Aroussi provides a Python library that connects to the Yahoo Finance API. The Quantstat library also provides a ready-to-go tearsheet with plots.

See the repo: https://github.com/ranaroussi/quantstats.

Google Finance

This is another popular API that is harder to access now. The Google Finance API is now only available through Google Sheets. Type the following into a cell: =GOOGLEFINANCE({SYMBOL}). Replace {SYMBOL} with a ticker symbol to get the current price of a market traded asset.

Stooq Pandas Datareader Interface

Surprised? Stooq has an API via a Pandas datareader interface that you can use to access daily, weekly, monthly, quarterly and yearly data. See https://pandas-datareader.readthedocs.io/en/latest/readers/stooq.html.

Other Pandas Datareaders

Need to refresh intraday? Check out the Pandas datareader list. The data sources here can be accessed programmatically using the Pandas datareader matching the data source listed. There is example code given here https://pandas-datareader.readthedocs.io/en/latest/remote_data.html.

Here are some Pandas datareader interfaces.

The Ever Changing World of Data

The world of financial data is constantly changing. Quantopian provided an online community where quants could practice and improve their skills. Then, as of November 2020, Quantopian is no more and many of the people behind Quantopian are now working at Robinhood! Pyfolio and Zipline continue to be available on Github but they don’t seem to be actively supported anymore. So, why do I mention this? The data sources listed above may work and they may not work depending upon when you’re reading this (I am writing this in December 2020, so this is my effort at future proofing this article!).

Code Sample for Stooq

By installing pandas and the pandas_datareader packages from Python’s pip, you can easily get data from Stooq using the following code snippet.

# requires ...
# pip install pandas pandas-datareader
import pandas as pd
import pandas_datareader.data as web
df = web.DataReader('AAPL', 'stooq')

Exploring Alpha Vantage’s API

We’ll utilize the pandas datareader for AlphaVantage in this next section. The pandas datareader for AlphaVantage can return:

  • Historical time series data (for daily, weekly, and monthly frequencies 20+ years is available)
  • Real-time stock quotes for up to 100 symbols
  • Real-time currency exchange rates (for physical and digital currencies)
  • Sector performance

An API key from AlphaVantage is required to use this datareader. API keys can be requested from this link: https://www.alphavantage.co/support/#api-key

For more information about the datareader go to this link: https://pandas-datareader.readthedocs.io/en/latest/remote_data.html#remote-data-alphavantage

The sector performance endpoint provides the percentage returns based upon companies in the S&P 500 for each sector:

                          RT      1D    5D        1M       3M       YTD     1Y      3Y       5Y      10Y
Energy                   1.13%  0.93%   -1.77%   -2.09%    12.97%   25.37%  40.00%  -33.75% -28.25% -38.86%
Financials               1.04%  1.85%   -0.02%    5.13%    16.64%   19.89%  64.47%  27.03%   85.10% 171.13%
Materials                0.86%  1.68%    0.37%    6.62%    11.19%   14.30%  69.65%  41.06%   75.60% 106.79%
Real Estate              0.57%  0.58%    2.04%    8.93%    15.14%   15.86%  27.59%  41.54%    
Industrials              0.44%  1.07%    0.43%    5.62%    14.73%   14.56%  67.60%  35.11%   75.18% 163.80%
Information Technology   0.35%  1.44%   -0.46%    8.14%     7.12%    9.36%  63.03%  117.00% 246.72% 491.92%
Communication Services   0.19%  1.11%   -0.63%    3.22%    10.71%   12.81%  54.77%  66.84%   51.50%  90.55%
Consumer Discretionary  -0.06%  1.03%   -1.23%    6.41%     3.97%    9.21%  56.98%  73.58%  125.48% 351.00%
Health Care             -0.29%  0.64%    1.77%    6.73%     5.07%    8.79%  25.65%  51.66%   73.35% 263.60%
Utilities               -0.45%  -0.17%  -0.95%    7.91%     5.83%    6.03%  12.89%  32.66%   40.75% 106.62%
Consumer Staples        -0.84%  -0.16%  -0.45%    4.24%     6.63%    2.55%  18.67%  37.89%   34.30% 123.97%

Since companies within a sector may perform similarly this information might help to identify companies to research. The sector performance data provides information regarding short-term (real-time) and long-term (10 year) sector performance. For example, Energy has had a negative outlook from 3 years to 10 years (-33.75% to -38.86%) but has recently begun to see positive returns year to date returns are 25.73% and 1 year returns are 40.00%.

import os
import pandas as pd
import pandas_datareader.data as web
df = web.get_sector_performance_av(api_key='xxxyyyzzz')

Company Fundamentals

AlphaVantage also supports retrieving company fundamentals in json format. Here is the example query string: https://www.alphavantage.co/query?function=OVERVIEW&symbol=IBM&apikey=xxxyyyzzz

{
    "Symbol": "IBM",
    "AssetType": "Common Stock",
    "Name": "International Business Machines Corporation",
    "Description": "International Business Machines Corporation provides integrated solutions and services worldwide.  ... The company was incorporated in 1911 and is headquartered in Armonk, New York.",
    "CIK": "51143",
    "Exchange": "NYSE",
    "Currency": "USD",
    "Country": "USA",
    "Sector": "Technology",
    ...
    "EPS": "5.98",
    "RevenuePerShareTTM": "82.739",
    "ProfitMargin": "0.0728",
    "OperatingMarginTTM": "0.1308",
    "ReturnOnAssetsTTM": "0.04",
    "ReturnOnEquityTTM": "0.2536",
    "RevenueTTM": "73782001664",
    "GrossProfitTTM": "35575000000",
    "DilutedEPSTTM": "5.98",
    "QuarterlyEarningsGrowthYOY": "-0.192",
    "QuarterlyRevenueGrowthYOY": "0.009",
    "AnalystTargetPrice": "141.3",
    "TrailingPE": "23.8177",
    ...
}

This query to the API endpoint returns a wide variety of information for a single company, including technical indicators such as moving averages. The Sector and Industry are also returned, so that this information can be used to group the company with other companies in the same industry and sector. As noted above under Sector Performance, identifying whether a company outperforms or under performs their sector can be useful in financial research. In the example below, IBM is considered to be in the Technology Sector. The Technology Sector, in general, has had positive returns. An interesting analysis might be to assess how IBM’s performance compares to the overall sector.

The API returns a JSON string that needs to be parsed into a Python data object.

def api_fundamentals(symbol, api_key):
    # Fundamentals API
    url = 'www.alphavantage.co'
    querystring = '/query?function=OVERVIEW&symbol={0}&apikey={1}'.format(symbol, api_key)
    http = urllib3.PoolManager()
    r = http.request('GET', 'https://'+url+querystring)
    data = json.loads(r.data.decode('utf-8'))
    df = pd.DataFrame.from_records(data, index=[symbol])
    return df

Leave a Reply