Chris Conlan

Data Scientist

  • About
  • Blog
    • Business Management
    • Programming with Python
    • Programming with R
    • Automated Trading
    • 3D Technology and Virtual Reality
  • Books
    • Fast Python
    • Algorithmic Trading with Python
    • The Blender Python API
    • Automated Trading with R
  • Snippets
  • Opportunities

Download Price History for Every S&P 500 Stock with R

May 16, 2017 By Chris Conlan 28 Comments

Update #9: Quandl API is Deprecated

According to an email I got from Quandl (and a few commenters corroborating), the Quandl EOD data API is no longer supported and is not providing data past March 27th. According to the CIO of Quandl, it was being provided for free by a 3rd party. That 3rd party is no longer providing the data, forcing us to search for other options.

Update #8: Yahoo! Finance API Back Up

The plot thickens once again! According to QuandMod’s getSymbols() function, the Yahoo! Finance API is currently the best free API that does not require registration. The URL endpoint of the Yahoo API, and the steps required to make a single download, have lengthened considerably.

Writing out the code required make a single download the new Yahoo API is more of a lesson in networking and cookie management than trading, so I will refrain from publishing it for the moment.

Switch to Quandl for Less Drama

In better news, Quandl has created an excellent API for downloading daily stock data. Their daily stock API is sort of a “free trial” for their premium subscription products. I spoke to the Quandl team about using it for frequent downloads and educational purposes, and they were fully supportive. So, I urge readers to stop depending on the fickle free services from tech giants, and sign up for an account and API key with Quandl.

Credit to GitHub user johnatasjmo for this solution:

# Quandl package must be installed
library(Quandl)

# Get your API key from quandl.com
quandl_api = "MYAPIKEY"

# Add the key to the Quandl keychain
Quandl.api_key(quandl_api)

quandl_get <-
function(sym, start_date = "2017-01-01") {
    require(devtools)
    require(Quandl)
    # create a vector with all lines
    tryCatch(Quandl(c(
        paste0("WIKI/", sym, ".8"),  #  Adj. Open
        paste0("WIKI/", sym, ".9"),  # Adj. High
        paste0("WIKI/", sym, ".10"), # Adj. Low
        paste0("WIKI/", sym, ".11"), # Adj. Close
        paste0("WIKI/", sym, ".12")), # Adj. Volume
        start_date = start_date,
        type = "zoo"
        ))
}

Update #7: Google Finance API Down

Because everything I write about breaks, the Google Finance API is now only delivering the most recent year’s data, regardless of what parameters are passed to it.

Update #6: Google Finance API is Hidden but Live

After more research, I discovered that R’s quantmod package is using a hidden version of the Google Finance API that works just fine. Below is our old Yahoo! Finance function re-written to support the Google Finance API. It is actually shorter and more easily read.

# Make sure data.table is installed
if(!'data.table' %in% installed.packages()[,1]) install.packages('data.table')

# Function to fetch google stock data
google <- function(sym, current = TRUE, sy = 2005, sm = 1, sd = 1, ey, em, ed)
{
  
  if(current){
    system_time <- as.character(Sys.time())
    ey <- as.numeric(substr(system_time, start = 1, stop = 4))
    em <- as.numeric(substr(system_time, start = 6, stop = 7))
    ed <- as.numeric(substr(system_time, start = 9, stop = 10))
  }
  
  require(data.table)
  
  google_out = tryCatch(
    suppressWarnings(
      fread(paste0("http://www.google.com/finance/historical",
                   "?q=", sym,
                   "&startdate=", paste(sm, sd, sy, sep = "+"),
                   "&enddate=", paste(em, ed, ey, sep = "+"),
                   "&output=csv"), sep = ",")),
    error = function(e) NULL
  )

  if(!is.null(google_out)){
    names(google_out)[1] = "Date"
  }
  
  return(google_out)
}

# Test it
google_data = google('GOOGL')

Downloading All S&P 500 Stocks with Google

I believe I encountered some rate limiting when I attempted to do this, but it works nonetheless.

# Load list of symbols (Updated May 2017)
SYM <- as.character( read.csv('http://trading.chrisconlan.com/SPstocks_current.csv', 
                              stringsAsFactors = FALSE, header = FALSE)[,1] )


# Hold stock data and vector of invalid requests
DATA <- list()
INVALID <- c()

# Attempt to fetch each symbol
for(sym in SYM){
  google_out <- google(sym)
  
  if(!is.null(google_out)) {
    DATA[[sym]] <- google_out
  } else {
    INVALID <- c(INVALID, sym)
  }
}

# Overwrite with only valid symbols
SYM <- names(DATA)

# Remove iteration variables
rm(google_out, sym)

cat("Successfully download", length(DATA), "symbols.")
cat(length(INVALID), "invalid symbols requested.\n", paste(INVALID, collapse = "\n\t"))
cat("We now have a list of data frames of each symbol.")
cat("e.g. access MMM price history with DATA[['MMM']]")

You should now be able to access the historical data of any symbol in the S&P 500 using, for example, DATA[[‘MMM’]] for 3M.

Notes on Performance and Usage

  • This downloads much faster than the old Yahoo! Finance API
  • I believe I encountered rate limiting after downloading about 200 symbols consecutively. This is not necessarily bad, it just means Google is doing a good job of controlling throughput. This is evidence that the API is better-constructed and less likely to be torn down. I found no documentation pointing to issues with multiple requests.
  • I have found evidence of a Google representative stating that this API is not to be used in the backend of any public application. I am curious about whether or not I will be able to use this code in Automated Trading with R in light of this statement. I believe it will not be a problem since quantmod has been publicly using it for so many years.
  • The data downloaded here is very similar to that used in Automated Trading with R. I believe the book can be adapted to use a service like this, although it will necessarily be less complex. There is a good amount of information missing from this historical data.
  • Volume seems to be missing from a good amount of symbols, and all prices are returned adjusted in advance. This is helpful computationally but removes a lot of implicit information about dividends and splits.
  • This API seems to download a strict maximum of 15 years of historic data. This is plenty of data, but I did have to switch the default start date up from 2000 to 2005 to accommodate this.
  • All of Google API’s are good about accepting a variety of date formats (and address formats for Maps), and this is no exception. As long as the dates are in month-day-year order with some delimiter like a dash, space, or plus sign, Google will interpret them correctly.

Altogether I think this a good replacement for Yahoo! Finance in its ability to provide free data for the purpose of retail trading and education. The trouble is, Yahoo’s API was hidden but still well-known. This API is hidden but does not seem to be very well-known. I will be making a post dedicated to this API for both R and Python in the future with the hope of making it the new educational standard for trading data.

Python Code

See this post: Download Historical Stock Data with Python for the equivalent code in Python.

Update #5: Google Finance API is Dead… Maybe?

I am not sure when this happened, but the Google Finance API is also dead. I know this must have been recently, because R’s quantmod package used to rely on this as its primary data source. I will keep searching for good free solutions and update this post with what I learn.

This is This is confusing to me, because R’s quantmod can still use Google as a source for historical price data. For example…

library(quantmod)

# This fails because it defaults to Yahoo
getSymbols("GOOG")

# This fails because it specifies Yahoo
getSymbols("GOOG", src = "yahoo")

# This works, even in light of the above message
getSymbols("GOOG", src = "google")

So, quantmod is still using Google as a source of data even though the finance API is dead, and has been since 2011 according to the above webpage. I will have to dive into quantmod’s source to figure out what is going on here. Updates to come.

Update #4: Yahoo! Finance API is Dead

After much discussion over the last two months, it is safe to say the Yahoo! Finance API will not be returning.

The question we are all trying to answer is, “What do we use now?” At this juncture, the answer will be different depending your means and goals. A few common situations are detailed below.

As an Author: I need a functional, comprehensive, and free API to support my book, Automated Trading with R. I am leaning towards Google Finance, and the other API’s that support the popular R package quantmod. The quantmod package built in a few fail-safes to Yahoo! Finance while still primarily relying on it, and seems for the most part unaffected by the outage. I expect to make some sacrifices regarding the volume and breadth of data while reworking the book, but these are welcome sacrifices in the name of accessibility and education.

As a Trader: I am comfortable paying premium rates for reliable data. Users in the comments of this post have suggested a handful of good options. There are a lot of low-cost services that attempt to be drop-in replacements for Yahoo! Finance, and there are a few high-value services that offer much more than Yahoo! Finance users are used to. Quandl, Bloomberg, and Reuters are examples of these high-value services, which, while costing significantly more, are very easy to scale and worth learning.

As a Researcher and Open-Source Developer: I feel that the Yahoo! Finance API’s death has destroyed a lot of opportunity for creating and sharing reproducible research. While researchers are not allowed to copy and disseminate Yahoo! Finance data, we are allowed to publish research that points to it. In other words, when we publish code, we include the download script in the beginning to allow “sharing” of financial time series data. It is an approach that gives Yahoo! credit where it is due, and still promotes sharing and learning. Now that Yahoo! Finance is gone, there is no standard approach to sharing financial time series data. I hope to make some posts in the near future to standardize a new free API (Google, Bloomberg, etc.) across popular languages.

In summary, we will rebuild.

Update #3: Yahoo! YQL API for Finance Data is also Affected

With hope for a new API but not clear end in sight, I got to work building a YQL-only solution for downloading historical price data. This was to serve as a backup in case we ultimately got no clarity on the CSV API. I discovered fairly quickly that the YQL API for accessing stock data, as it is popularized on the internet and as it is expressed in Automated Trading with R, depends internally on the http://ichart.yahoo.com API and is therefore inaccessible.

In the same way that we see “Could not process this ‘GET’ request” on the CSV API, the YQL API simply returns a JSON list of all of the stocks requested, each with its own individual “Could not process this ‘GET’ request” message. So, the YQL API itself is not down or unhealthy, rather, it is doing its job very well by alerting us that each individual stock is inaccessible.

Update #2: Yahoo! CSV API is Live but Undocumented

As of May 27th, 2017, we are seeing movement in the Yahoo! Finance CSV API. It appears not all hope is lost. Calls to https://ichart.yahoo.com no longer return “We are working on it” messages, rather, they return “Could not process this request” messages.

What we appear to have now is a functional but totally undocumented API.

The facts at this point:

  • All requests, like GET, POST, PUT, and DELETE, all return a message along the lines of “Description: Could not process this request.”
  • URLs are still being auto-forwarded to HTTPS
  • We no longer have a “Coming Soon” or “Working on it” message
  • API Calls using the old parameters return this error message, even with alternate HTTP request types.
  • I cannot find new documentation on this API

I have emailed Yahoo! about this and hope to receive a response soon. I am pleased that my original prediction of receiving a restructured API seems to be correct, as this will necessitate fewer and less drastic changes to Automated Trading with R.

Please comment below if you have any information to share that is not presented in this post.

Update: Yahoo! API Inactive and Pending Changes

As my luck would have it, the Yahoo! Finance API at ichart.yahoo.com went down within hours of me making this post. As of the afternoon of May 16th, 2017, the API calls in the script below will not download anything meaningful. Fortunately, Yahoo! seems to be making some structural changes and improvements to the API.

If you enter an API call into your web browser, you will see this splash screen:

The Facts

Here is what we know so far:

  • Countless open-source software packages and educational resources depend on the free Yahoo! Finance API
  • Yahoo! has made no official statement about rescinding public access to this data.
  • The API has been called “secret” and “hidden” because it is dificult to find official Yahoo! documentation on it. It has been mostly taught and passed along by knowledgeable professionals and educational resources.
  • The splash screen says “Our engineers our working quickly…”
  • All calls to ichart.yahoo.com are auto-converting to HTTPS regardless of their validity.
  • Calls to ichart.yahoo.com are setting new cookies that we haven’t seen before.

My Prediction

I am inclined to believe that Yahoo!’s engineers are indeed working to resolve the issue. As I have watched this situation evolve, I have seen the endpoint die, then get a splash screen, then get hooked up to SSL, then set new cookies. All of this points to a significant restructuring effort.

I believe the Yahoo! Finance API will update the structure of its calls, publish some legitimate documentation, and begin requiring authentication to access it. Hopefully that happens quickly, and I can update the scripts below. I will be watching vigilantly for this update, and likely publish a new edition of Automated Trading with R upon its release.

Original Post: Downloading Price History with R

In Automated Trading with R, we build a complete automated trading platform using the R language. In the second chapter, we get 15+ years of daily price data on every stock in the S&P 500 loaded into R using free API’s. The code required to do this is surprisingly brief and straightforward.

Thanks, Yahoo!

I have copied an R script below that will load historical price data of every S&P 500 right into R using the Yahoo! Finance API. In the book, we expand on this, explain how it works, and continue to refine this data.

Hopefully this script will serve as a quick solution to systems traders searching for a significant and reliable data source.

# Load list of symbols (Updated May 2017)
SYM <- as.character( read.csv('http://trading.chrisconlan.com/SPstocks_current.csv', 
                              stringsAsFactors = FALSE, header = FALSE)[,1] )

# Make sure data.table is installed
if(!'data.table' %in% installed.packages()[,1]) install.packages('data.table')

# Function to fetch yahoo data
yahoo <- function(sym, current = TRUE,
                  a = 0, b = 1, c = 2000, d, e, f,
                  g = "d")
{
  
  if(current){
    system_time <- as.character(Sys.time())
    f <- as.numeric(substr(system_time, start = 1, stop = 4))
    d <- as.numeric(substr(system_time, start = 6, stop = 7)) - 1
    e <- as.numeric(substr(system_time, start = 9, stop = 10))
  }
  
  require(data.table)
  
  tryCatch(
    suppressWarnings(
      fread(paste0("http://ichart.yahoo.com/table.csv",
                   "?s=", sym,
                   "&a=", a,
                   "&b=", b,
                   "&c=", c,
                   "&d=", d,
                   "&e=", e,
                   "&f=", f,
                   "&g=", g,
                   "&ignore=.csv"), sep = ",")),
    error = function(e) NULL
  )
}


# Hold stock data and vector of invalid requests
DATA <- list()
INVALID <- c()

# Attempt to fetch each symbol
for(sym in SYM){
  yahoo_return <- yahoo(sym)
  
  if(!is.null(yahoo_return)) {
    DATA[[sym]] <- yahoo_return
  } else {
    INVALID <- c(INVALID, sym)
  }
}
# Overwrite with only valid symbols
SYM <- names(DATA)

# Remove iteration variables
rm(yahoo_return, sym)

cat("Successfully download", length(DATA), "symbols.")
cat(length(INVALID), "invalid symbols requested.\n", paste(INVALID, collapse = "\n\t"))
cat("We now have a list of data frames of each symbol.")
cat("e.g. access MMM price history with DATA[['MMM']]")

Save Data to CSV Files

After running the above, run the following to save each stock’s price history as a CSV file to a folder in your working directory.


# Folder where you are storing the data
setwd('~/Desktop/stockdata')

# 
for( sym in names(DATA) ){
  write.csv(DATA[[sym]], paste0(sym, '.csv'), row.names = FALSE)
  cat('Wrote', sym, 'data to CSV file.\n')
}

Navigate to the folder you set above to see hundreds of CSV files.

See my study of the S&P 500’s long-term behavior for some analysis on this type of data.

Bonus: Golfing the Download Script

To showcase just how efficient the R Language can be, we have condensed the first script from this post into under 400 characters (about 3 tweets). There is less error checking, and we do not make use of the more efficient data.table package, but the list D is more or less the same as DATA from the first two examples.

l='http://trading.chrisconlan.com/SPstocks_current.csv'
S=read.csv(l,header=FALSE)[,1]
f=function(s){
  b=c(0,1,2000)
  i=list(c(6,7),c(9,10),c(1,4))
  v=c(b,sapply(i,function(v)substr(Sys.time(),v[1],v[2])),'d')
  v[5]=as.numeric(v[5])-1
  read.csv(paste0("http://ichart.yahoo.com/table.csv?s=",s,paste0('&',letters[1:7],'=',v,collapse=''),'&ignore=.csv'))}
D=list()
for(s in S)D[[s]]=f(s)

Note: Check that ichart.yahoo.com is Healthy

If your code is hanging, visit ichart.yahoo.com and check to make sure it is live. It has been experiencing occasional outages.

Filed Under: Automated Trading, Programming with R

Comments

  1. Nick page says

    May 17, 2017 at 2:48 am

    Yahoo URL no longer works. I believe it’s a permanent change.

    Reply
    • Chris Conlan says

      May 17, 2017 at 5:16 pm

      Hi Nick,

      See my update above. Please let me know if you have any other sources on the issue.

      Reply
  2. Jens says

    May 17, 2017 at 6:54 am

    Hey Chris,
    I found your blog post while I was searching the web for updated Information on how to download hist. prices from Yahoo.
    The problem is: Yahoo changed the URL which let me access hist. data in the past. It did not change just from http to https (as others suggest). It changed the access parameters too. From the download link they provide next to the historical data of any stock I cannot really derive how the parameters should look like to put that all in a script like you did obove. Any idea how to sole this?
    Cheery, Jens

    Reply
    • Chris Conlan says

      May 17, 2017 at 5:20 pm

      Hi Jen,
      See my update above.
      Yahoo! has always had a handful of methods for fetching stock data. My book uses both this CSV API and the YQL API concurrently. If you are referring to downloads like this:
      https://finance.yahoo.com/quote/AAPL/history?p=AAPL
      I don’t believe that can be efficiently collected.
      Hopefully we get an update soon.

      Reply
  3. Denis says

    June 5, 2017 at 4:44 pm

    Hi,
    What are the news here?
    From I’ve found, it seems that Yahoo! decided to protect their historical data from access by robots or scripts. I found they check the presence of cookies so that history can only be fetched one at a time from a web browser.
    Did you make progress on your side?
    Any other source of historical data than Yahoo! ?
    D.

    Reply
    • Chris Conlan says

      June 5, 2017 at 8:42 pm

      You may be right on that front. I have no new updates on the ichart.yahoo.com end. I haven’t heard back from any of their help lines. I am thinking we need to try Google stock data.

      The R Package “quantmod” used to depend on the Yahoo finance API but no longer does. I will check that source code to see where to go next.

      Reply
      • Andi says

        July 3, 2017 at 5:26 am

        Hi Chris

        I have bought your book Automated Trading with R, now I have see the problems with the Yahoo Finance APIs. Is it possible to provide other codes, for example, to fetch Google stock data?

        Thanks for an answer, really appreciate.

        best regards,
        Andi

        Reply
        • Chris Conlan says

          July 4, 2017 at 3:59 pm

          Hi Andi,
          Thanks for picking up the book. I am working with my publisher to push a second edition that relies on a new API. I have yet to find a service that correctly balances breadth of data and low barriers to use, but I am leaning towards Google. Will post updates here.
          Thanks,
          Chris

          Reply
  4. Denis says

    June 5, 2017 at 4:45 pm

    Check this out:

    $ curl -o test ‘https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1483225200&period2=1496680048&interval=1d&events=history&crumb=X1DyIFBT2O2’

    $ cat test
    {
    “finance”: {
    “error”: {
    “code”: “Unauthorized”,
    “description”: “Invalid cookie”
    }
    }
    }

    Reply
    • Chris Conlan says

      June 5, 2017 at 8:36 pm

      I’ve never tried that API endpoint. When I try entering that URL you left on a browser I get the same message.

      Reply
  5. Bob says

    June 9, 2017 at 11:24 pm

    We need to come together and figure out a permanent solution to this. Everybody needs to think about an R solution that works for equities / currencies / mutual funds…everything that yahoo used to offer with adjusted close. This data was like oxegon to all of us, and yahoo failed us yet again. Should we get congress involved? FDR (Financial data reform ) sounds good to me.

    Reply
    • Max says

      June 12, 2017 at 9:20 pm

      Well, I guess we can demand free data, but only if …….sorry, can’t come up with any good reason why anyone’s entitled to anything free in corporate world. Has anyone used https://www.quandl.com/?

      Reply
      • Denis says

        July 1, 2017 at 7:00 pm

        Quandl is good, but a little bit expensive for personal use. I switched to https://eodhistoricaldata.com, much cheaper and data is good.

        Reply
  6. Jens says

    June 28, 2017 at 8:56 am

    Last month I found Alpha Vantage: https://www.alphavantage.co/. They provide historical prices of US stocks up to 20 years (incl. adjusted prices, as many folks seem to like it) and real time quotes of international stocks. They even provide intraday minute based charts for US stocks. Everything in JSON format.

    I discovered the service some days after the “Yahoo desaster”. Their adjusted prices offering was announced just two or three weeks ago. So this makes me belive they are actively developing their platform.

    Access is easy. Get their API key and start pulling their data. I’ve been testing for some time now with PHP-scripts and JS. What I found so far is:

    1) Pulling historical data works a little slower compared to the old Yahoo API, but it’s sufficent for my needs. Real time quotes work fine. However, I did not check wether it is really real time or the quotes they provide are correct.

    2) Sadly some historical chains ommit some days. Others provide open, high, low, close but leave out volume for a handful of days. I found that, for example, for MO and PM. In the beginning of my tests, these quote chains were not corrupted, now they are. I hope that will change again.

    3) Intraday minute charts are really cool. You did’n get that from Yahoo. These charts cover several days depending on the interval. Interval means minute, 5-minute, 15-minute etc.

    4) They even provide indicators and moving averages. I did not try these as I render all this stuff on my local machine using the raw OHLC-data.

    My main focus so far was setting up the scripts and get them to work. It’s an https-call, so my server needed some adjustments to let get_file_contents(‘https://….’) do its job as intended. However, with a local server on my Mac (MAMP installed, nothing changed) everything worked great from the start. I tried to implement caching, so that I do not need to pull the whole chain with every call. (All this PHP programming is the tricky part for me as I’m not the biggest friend of PHP…)

    As soon as I discover anything new I’ll let you know.
    Thanks Chris for keeping this blog thread alive.

    Jens

    Reply
    • Denis says

      July 1, 2017 at 7:01 pm

      Unfortunately, I didn’t find on Alphavantage a lot of funds, then it doesn’t work for me.

      Reply
    • joshua laferriere says

      July 29, 2018 at 11:21 am

      interpolate your missing data using zoo my friend 🙂

      Reply
  7. Denis says

    July 1, 2017 at 6:59 pm

    I’ve tested several alternatives and found that https://eodhistoricaldata.com the best one for those who used Yahoo Finance. The services provides raw data, adjusted closes and splits/dividends. They also have CSV output, with very similar format for Yahoo Finance users.

    The biggest advantage of eodhistoricaldata.com in that they have very similar API to Yahoo! Finance, then it’s very easy to switch to them.

    Reply
  8. Harry Espino says

    July 6, 2017 at 7:07 am

    I have moved to MarketXLS after this change, much more reliable data.

    Reply
  9. Chris Conlan says

    July 24, 2017 at 5:18 am

    For everyone following this comment thread, I have added documentation for the hidden Google Finance API at the top of the post. It seems to be a great replacement for Yahoo! Finance.

    Reply
    • Azadeh Fereidoni says

      October 17, 2017 at 8:41 pm

      Hi Chris,

      I am trying to get data for some TSX like BMO using your code and I was not able to fetch Google data. It works for GOOGL but not BMO. can you give me some hints?

      Thanks in advance.
      Azadeh

      Reply
  10. Kelvin says

    August 28, 2017 at 6:54 pm

    Hi Chris,

    By using the google function you provided, I am able to get csv files downloaded and store them in the stockdata folder. However, when I run code 2_5 to read csv file by fread, some variables such as Date, Open and High end up being defined as character data type. I tried to modify the code but it still does not work

    ===========================================
    require(data.table)
    fread("MMM.csv", na.strings=c("-", "N/A", "NA", "", " "), sep = ",", colClasses = c("Date" = "Date", "Open" = "numeric", "High" = "numeric", "Low" = "numeric", "Close" = "numeric", "Volume" = "integer"))
    ==========================================

    Could you give me some hints how I can get this resolved?

    Thank you in advance

    Kelvin

    Reply
    • Chris Conlan says

      August 28, 2017 at 11:53 pm

      Hi Kelvin,

      I can’t be sure of how to help you without seeing the CSV files you are attempting to read, but I can give a few alternative options.

      1) Try using read.csv() instead. This function is a little slower, but better at auto-detecting data types.
      2) Manually change the class after you have read in the data through fread(). It can look like DATA[[‘MMM’]]$Open = as.numeric(DATA[[‘MMM’]]$Open).
      3) Try specifying the colClasses argument of fread() as an atomic vector instead of a named vector.
      4) Diagnose the error better by removing the supressWarnings() wrapper.

      Thanks for picking up the book.
      Hope this helps,
      Chris

      Reply
  11. Jens Blaustein says

    October 2, 2017 at 3:10 pm

    Hi Chris,

    Dennis Lee on GibHub provides a nice solution which makes the Yahoo historical quotes available again. I found it some weeks ago after it was published. Here’s the link: https://github.com/dennislwy/YahooFinanceAPI

    The solution is built using C# and the .NET-Framework. As I’m a C# coder for years I tried it out and it works as it did before the API change. A good work!

    The C# classes provided scrap the cookie and a so called crumb from a call to a random Yahoo quote page. The cookie is read out from the header and the crumb somewhere from within the same site’s source code. The eventual request for historical quotes embeds the cookie in its header and adds the crumb to the end of the URL. Done!

    The cookie and crumb combination only works when both the cookie and the crumb are scraped from the same page. Otherwise the eventual quote request returns an error. The cookie and the crumb can be stored an used again for later requests. They seem to be valid for a certain time. Updating them once in a day is probably enough.

    I developed a PHP version from Dennis Lee’s solution. This also works perfektly. So…. Yahoo Finance historical quotes are back! At least if your have a (local) server for PHP or you are able to code some C#.

    In an earlier comment I mentioned that I switched to AlphaVantage. It provides historical quotes for free as well and, seemingly, in a good quality. However, the service is slow slow slow. Sometime requests for only one price history take 20 seconds or longer. Yahoo Finance is spot on with almost no delay.

    Jens

    Reply
    • Chris Conlan says

      October 2, 2017 at 3:15 pm

      Well, this is certainly cool news! Thanks for sharing. I’ll have to look into the terms of use on this one to make sure we aren’t botting where we shouldn’t. At the very least, it is pretty inventive code.

      There is a Python version of what you referenced as well: https://github.com/c0redumb/yahoo_quote_download

      Reply
  12. Alex Euler says

    October 11, 2017 at 5:36 am

    Hi Chris,

    Thanks for the great info here. I’m just getting into this subject, and while getting historical price data would be great, I already have my eye on the issues arising from updating that data as time goes forward. What’s your preferred method for updating price data stored locally? I’ve thought of using a function to add rows to spreadsheets corresponding to (Current T) – (Last Updated T) and re-save spreadsheets. However, I’m skeptical that this is the best or most efficient way, especially when there are APIs out there claiming to give real-time ticker information. (Not that this is automatically a life saver if the goal is to get the data locally stored, which is pretty much a must for any computational work.) Any thoughts?

    Reply
  13. Jens says

    November 2, 2017 at 8:17 pm

    Hey folks,

    Yahoo just shut down another (well, unofficial) API that was widely used by thousands of traders for years: the download of intraday quotes via http://finance.yahoo.com/d/quotes.csv?….

    Here’s the link with a confirmation: https://yahoo.sdx.socialdynamx.com/portal/conversation/19248672

    Now there is no reliable free API for fetching current quotes of stocks, indices or currencies anymore – at least that I know of.

    Jens

    Reply
  14. steven says

    January 18, 2018 at 1:47 am

    Hi Chris,

    I am tracing your code (kind of late to see the sim results since neither Yahoo nor Google APIs is working now) and trying to make the code in the book (and github) work so that I can learn. However, due to the data object change in Quandl, compared to Yahoo’s API, I see lots of places need to be update. Specifically, you had already pointed out one in one of your comments:

    2) Manually change the class after you have read in the data through fread(). It can look like DATA[[‘MMM’]]$Open = as.numeric(DATA[[‘MMM’]]$Open).

    So originally DATA[[“Close”]] needs to be changed to DATA[[‘MMM’]]$Close along with DATA object type changes (I need to tweak the zoo object to fit in Quandl return object to at least run some code, not sure if this is the right way), so simulation lists like 7-2.R just don’t work anymore.

    So my question to you is, do you see a quick way to plug in the Quandl data object in your original code, for example, make List 7-2 (one of the examples) work? Or do you have other plans to publish new code based on Quandl?

    Thanks

    Reply
  15. Timothy Nessus says

    November 23, 2018 at 4:01 am

    Hi, I have been with http://eodhistoricaldata.com/ for a few months now and I had extensive contact with their support people. This is the result:
    PROS:
    – They are cheap
    – They are kind of complete
    – Their history goes back sufficiently to do backtesting
    CONS:
    – They are amateurs. Their starter code is not theirs and it is full of bugs. You better know how to program.
    – Their help pages are not very helpful and they make a ton of assumptions which may or many not be true or may or may not be applicable or may or may not be understandable. Obviously these people never wrote a technical document, much less a User Requirement Specification / System Requirement / Design Specification, etc. (if you are into software engineering).
    – Some exchanges may “disappear” on you without notice
    – Their support is OK, but it will take you a few tries to communicate with them; their english is not great (this is a French company).
    – Their API is average, but, they do not have useful “bulk” download options. If you want to download entire exchanges from scratch, you will have to do it one day at the time.
    – They are not sure when EOD data will be ready.
    – Data may not be available EOD for… no reason.

    BOTTOM LINE: if you want cheap data to develop systems and you need a lot of it, this is a good option. Pay for a month or two, download everything and then develop your systems. Once you start trading for real, get a reliable data provider.

    Reply

Leave a Reply Cancel reply

For Traders

Algorithmic Trading with Python by Chris Conlan

Available for purchase at Amazon.com.

Algorithmic Trading

Pulling All Sorts of Financial Data in Python [Updated for 2021]

Calculating Triple Barrier Labels from Advances in Financial Machine Learning

Calculating Financial Performance Metrics in Pandas

Topics

  • 3D Technology and Virtual Reality (8)
  • Automated Trading (9)
  • Business Management (9)
  • Chris Conlan Blog (5)
  • Computer Vision (2)
  • Programming with Python (16)
  • Programming with R (6)
  • Snippets (8)
  • Email
  • LinkedIn
  • RSS
  • YouTube

Copyright © 2021 · Enterprise Pro Theme On Log in