Ruby gem for stock / finance data

A post on the yahoo finance / stock gem had been overdue for a long time. Last week I added the xml support for the result formatting and published a new version of the gem. So I thought its a good time to write a bit about how I got started with it, implementation details, usage, etc.

Problem

My mother has a portfolio of stocks and it became a bit difficult for her to keep an eye on their performance either by using online or print medium. So she beautifully used her powers of delegation and asked me to check them regularly :). As usual, first thought, use google or yahoo finance for this kinda stuff but soon I wanted to generate more data for the stocks and I found that they lack them. I said, what the hell and built a rails application that would go to a stock exchange site and scrape the pages. Then I thought, an app like this could be useful for other people like me but the problem was, the site site I was using for scraping was country specific. I decided to make it a bit more general purpose before putting on the github and started using ScrAPI gem (ScrAPI uses CSS selectors for scraping) to scrape data from a site that has stocks from more than just one country. ScrAPI was working fine but wasn’t consistent in some cases. So I started looking for finance APIs, obviously google has a finance API but you need an account with google finance to use it. I think that’s a bit inconvenient for user’s to first have a google finance account to use a library though they may already have but that’s still a hassle. So started looking elsewhere and found yahoo finance. Best thing about yahoo finance is you don’t need any account to use it. And yahoo provides data as CSVs and its easy to use YQL (Yahoo Query Language). Hence, I decided to use Yahoo finance for the data purposes and write a ruby gem that can be used in any ruby or rails application. On another note, I met a couple of awesome yahoo guys at barcamplondon7 who showed me some cool APIs; Developer Apps like, open tables, YAP, blueprint etc,. Check them out here

This is how I got started on nas-yahoo_stock gem.

Implementation

The tests for the gem code are written using rspec 1.2.2 and it has cucumber features for integration testing.

There are different interface classes (based on their responsibilities) that are derived from the base interface class. For example, the quote interface provides quotes for multiple companies in a single request with variable parameters, symbol interface submits request to a yahoo page and scrapes data using some regex rules, etc.

Then we have different result classes to easily present result data in the format we want them. This has been achieved by implementing the Strategy Pattern to switch easily between different result formats. Benefit: a single result interface for the YahooStock::Quote / YahooStock::History / YahooStock::ScripSymbol classes for getting and presenting data into different formats (the examples are given in the Usage section).

I kept one thing into consideration while building the gem was to send requests to yahoo only when it is absolutely necessary. For instance, the gem sends a request to yahoo only when the results method is called on an object like YahooStock::Quote / YahooStock::History / YahooStock::ScripSymbol and not during any other operation. Which is fine but what about calling results multiple times for the same data and format or to generate different formats for the same data? An easy solution for that is to implement the Observer Pattern. The observers in nas-yahoo_stock gem keep an eye on the data requested whenever the results method is called on the quote, symbol or history objects and if the data requested currently is different from the previous request then only the gem hits yahoo else it returns the previous result without hitting yahoo.

The source code for the gem is available on github -> nas-yahoo_stock

Usage

To use the library, start by:

gem sources -a http://gemcutter.org
sudo gem install nas-yahoo_stock

Once the gem is installed, we can go to the irb and

require 'rubygems'
require 'yahoo_stock'

Before finding the history or quote information about a company stock we will need the stock symbol for that company. YahooStock::ScripSymbol class gives us all the possible stock symbols for a company. For instance, if we want to find the stock symbol for Yahoo.
symbol = YahooStock::ScripSymbol.new(‘Yahoo’)
To get the results,
symbol.results.output
the above method call give us a string of all the values.
If we want to store the result in a file and come back to it later, then we can
symbol.results.store('file_name')
The results can be easily presented into different formats like
1. to return an array instead of a string use:
symbol.results(:to_array).output
2. to return a hash, use:
symbol.results(:to_hash).output
3. to return xml, use:
quote.results(:to_xml).output

Once we have the stock symbol, then the quote or history information about that stock can be retrieved easily. Lets get to the history part first.

To get a stock’s history, the YahooStock::History class needs to be initialized with a hash of stock symbol, start and the end date for which we need the stock data

history = YahooStock::History.new(:stock_symbol => ‘yhoo’, :start_date => Date.today-20, :end_date => Date.today -2)
Then we can call all the same methods on the history object that we have called on the symbol object to get results, like
#to get history as an array
history.results(:to_array).output
#to return a hash, use:
history.results(:to_hash).output
#to return xml, use:
history.results(:to_xml).output
#to store results in a file, use:
history.results.store('filename')

To get quote information about a stock
1. Initialize quote object
quote = YahooStock::Quote.new(:stock_symbols => [‘YHOO’, ‘GOOG’])

If we call the results method on the quote object then it will give us only two values for the above two stocks which are included by default on object initialization. But a number of parameters can be passed to the quote object to get information on and these parameters can be viewed by sending valid_parameters message to the quote object.
quote.valid_parameters

To view the current parameters being used in the quote object
quote.current_parameters

The parameters are also categorised in realtime, standard and extended categories. To use parameters for these categories just pass one of the category names to the quote object and it will use those parameters, for e.g.
quote.realtime

To use all parameters just use:
quote.use_all_parameters

To view the current stock symbols used
quote.current_symbols

To add more stocks to the list
quote.add_symbols(‘MSFT’, ‘AAPL’)

To remove stocks from list
quote.remove_symbols(‘MSFT’, ‘AAPL’)

Then we can call all the same methods on the quote object that we have called on the symbol and history objects to get results, like
#to get quote as an array
quote.results(:to_array).output
#to return a hash, use:
quote.results(:to_hash).output
#to return xml, use:
quote.results(:to_xml).output
#to store results in a file, use:
quote.results.store('filename')

If you need the results in some other format apart from the formats provided by the gem then you can simply pass in a block to the result object like
quote.results{YourSpecialFormat.method_name}

32 thoughts on “Ruby gem for stock / finance data”

  2. Abhi says:
    Reply

    Hi Nasir,

    Thanks for providing such a great yahoo stock gem. It helped me a lot to get the stock quotes information from yahoo finance instead of scraping the contents from page. Can you please provide such a gem to get the data from google finance also. I have tried with the gdata ruby gem, but i am not able to use the google finace api using gdata gem. Can you please help me? Is there any way to get the data from google finance in and out of google gadget editor? I am able to get the data using gadget, but i am not able to get in the html page.

  3. aashish says:
    Reply

    Hi Nasir,

    I am developing a google finance interface using ruby for the past 2 weeks. I followed the link http://code.google.com/apis/gdata/articles/using_ruby.html#authentication
    I authenticated succesfully. Now I am confused about how to proceed further. Since you have worked on finance api’s, I think you can give better idea of using google finance api. To be more clear I don’t have any idea of stocks or finance in general. Can you please help me in explaining about stocks and finance and google finance data(how to get data and represent it on a desktop application using ruby?

  4. nasir says:
    Reply

    @abhi

    Thanks for the feedback Abhi and I am glad that you like the nas-yahoo_stock gem. Regarding writing a ruby library for google finance, yes that was on my list when I started writing yahoo_stock gem but later on other things took priority and it just kept sliding down on my priority list. So yes I have plans for it but currently I am trying to push a new release for url_shortener gem and then a new library that can get rss feed and post to twitter at pre-defined regular intervals. Once I have done these two things then I will probably look into the google finance gem.

    @ashish & @abhi

    I think the best starting point would be http://code.google.com/apis/finance/developers_guide_protocol.html that explains all the end points, parameters and the xml it returns. Then it is just a matter of grabbing the right xml element you want.

  7. Isaiah says:
    Reply

    I just wanted to note that if you want to get quotes for an index (IE S&P500, ticker symbol ^GSPC) that you have to use the hex code %5E (%5EGSPC) instead of the ^ because there is a bug in the URI package that causes a bad uri error if that character is present 😦

  8. nasir says:
    Reply

    Thanks Isaiah, I will fix it so that symbols get converted into their hex code before URI package uses them and commit it asap (probably in a day or two) so that you can use ^GSPC. I could have done it now but it is not possible to do from where I am right now 🙂

  9. ts says:
    Reply

    Have you considered a method for requesting Options chain data for a particular security? I’m really interested in analyzing options but it’s difficult to find tools that help retrieve that information.

  10. nasir says:
    Reply

    @ts I did not have options chain in mind when developing this gem as I mainly developed it to monitor stock prices. Does yahoo provides any API or way to download this data?

  13. Nemo says:
    Reply

    Nasir,

    Thanks so much for writing and publishing this!

    I’m just starting to play with it, so I thought I’d follow along in IRB.

    First, allow me to point out two typos in your example for history:
    history = YahooStock::History.new(:stock_symbols => ‘yhoo’, :start_date => Date.today-20, end_date => Date.today -2)
    Should be :stock_symbol (singular), and :end_date (not end_date), right?

    Now, when I try
    history.results(:to_hash).output

    I get this:
    history.results(:to_hash).output
    TypeError: wrong argument type Symbol (expected Proc)
    from C:/Ruby/lib/ruby/gems/1.8/gems/nas-yahoo_stock-1.0.4/lib/yahoo_stock/history.rb:27:in `data_attributes’
    from C:/Ruby/lib/ruby/gems/1.8/gems/nas-yahoo_stock-1.0.4/lib/yahoo_stock/base.rb:26:in `results’
    from C:/Ruby/lib/ruby/gems/1.8/gems/nas-yahoo_stock-1.0.4/lib/yahoo_stock/result/hash_format.rb:14:in `initialize’
    from C:/Ruby/lib/ruby/gems/1.8/gems/nas-yahoo_stock-1.0.4/lib/yahoo_stock/base.rb:26:in `new’
    from C:/Ruby/lib/ruby/gems/1.8/gems/nas-yahoo_stock-1.0.4/lib/yahoo_stock/base.rb:26:in `results’
    from (irb):13

    By the way, history.results(:to_array).output works beautifully…

    Any ideas?

    Thanks again,
    Nemo

  14. nasir says:
    Reply

    @Nemo thanks for pointing out the typo, thats fixed now.

    Regarding the error you are getting, I think its because of the use of symbol to proc in the code which I think was introduced in 1.8.7

    So probably you have older version of ruby installed. Can you please check which version of ruby you are using?

  16. antony says:
    Reply

    hi nazir

    i am very new to ruby on raills

    1.i have windows xp os and istalled ruby on rails 1.8.6

    2.i am developing yahoo finance interface past one week

    3.i was installed “gem install nas-yahoo_stock” through command prompt

    4.what should do to with require ‘rubygems’ and require ‘yahoo_stock’ and wat u mean IRB
    5.where we need to save source code of nas-yahoo_stock

    6.how to get the output value through command prompt

    7.while trying this following command
    [symbol = YahooStock::ScripSymbol.new(‘Yahoo’)] i got the error ‘symbol’ not recognized as an internal or external command

    please help me out to know this

    thanks and regars

  17. nasir says:
    Reply

    @antony

    I think you are using an older version of ruby that’s why you are getting the error

    ‘symbol’ not recognized as an internal or external command

    Please install ruby 1.8.7 or later then it should be fine.

    IRB stands for Interactive Ruby and is used to run ruby codes/scripts interactively. Since you are on windows, just open command prompt, type irb and hit the ENTER key. Then you can run any ruby code and get results.

    Regarding the source code, it has been installed in you ruby gem directory and doing require ‘rubygems’ and require ‘yahoo_stock’ makes it available to your environment.

    Hope that helps.

    • antony says:
      Reply

      hi nazir
      thanks for valuble information then according to your words
      1.i installed ruby 1.8.7 and installed the gem also

      2.after enter into IRB and write the
      require ‘rubygems’
      require ‘yahoo_stock’
      3.further what can i do how to get the result

      4.i want Write a small Ruby program that could be run from the command line. Two parameters are given, the stock symbol, e.g. AAPL and the expiration year month as YYYY-MM, e.g. 2010-05. our program will go to the Yahoo finance page and calculate for call options total Open Interest by summing all the lines under Open Int for calls and put options total Open Interest for put options. please help me out

      • rajesh says:

        hi nasir

        I installed ruby1.9.1 and want to execute //nas-yahoo_stock// in command prompt using win xp os i followed your above given usage details

        I connfigured these following two things on my prompt c:/>ruby19>lib
        1. gem sources -a http://gemcutter.org
        2. sudo gem install nas-yahoo_stock

        then entered into IRB and written
        require ‘rubygems’
        require ‘yahoo_stock’

        I entered all steps in in IRB window. After I finished all steps i entered [symbol.results.output] nothing happen over there please help me out how to get the result

  19. nasir says:
    Reply

    @rajesh

    If you are not able to see any output then probably you might have missed a quote or something.

    Anyway, I have just pushed after sorting a ruby 1.9.1 compatibility issue in history interface class so currently the latest version of nas-yahoo_stock gem is 1.0.5

    Can you please check the what version of gem are you using? (by running ‘gem list nas-yahoo_stock’) and install the new version.

    Then follow the steps again. It worked fine when i tried just now. Below are some of the method calls in the console and their results you should see

    require ‘rubygems’
    #=> true

    require ‘yahoo_stock’
    #=> true

    symbol = YahooStock::ScripSymbol.new(‘Yahoo’)
    #=> #<YahooStock::ScripSymbol:0x1000144 @company="Yahoo", @interface=#<YahooStock::Interface::ScripSymbol:0x100011c @after_element="yfi_fp_left_bottom", @company="Yahoo", @observer_peers=[#], before_element”yfi_sym_results”, base_url”http://finance.yahoo.com/lookup/all”

    symbol.results.output
    #=> “CFYH.PK,CITIGROUPFDGYAHOO,10.49,Stock,,PNK\r\nYA-U.TI,YAHOO,12.75,Stock,,TLO\r\nYHO.HM,YAHOO,12.50,Stock,,HAM\r\nYHO.BE,YAHOO,12.51,Stock,,BER\r\nYHO.DE,YAHOO,12.50,Stock,,GER\r\nYHO.DU,YAHOO,12.48,Stock,,DUS\r\nYHO.F,YAHOO,12.29,Stock,,FRA\r\nYHO.MU,YAHOO,12.93,Stock,,MUN\r\nYHO.SG,YAHOO,12.51,Stock,,STU\r\nYHOO.MX,YAHOO,210.00,Stock,,MEX\r\nYHOO.BA,YAHOOINC.,7.50,Stock,,BUE\r\nYOJ.DE,YAHOOJAPAN,250.00,Stock,,GER\r\nYOJ.F,YAHOOJAPAN,304.60,Stock,,FRA\r\nYOJ.BE,YAHOOJAPAN,216.11,Stock,,BER\r\nYAHOF.PK,YAHOOJAPANCORP,318.58,Stock,,PNK\r\nYAHOY.PK,YAHOOJAPANCORP,121.75,Stock,,PNK\r\nYHOO,Yahoo!Inc.,15.29,Stock,InternetInformationProviders,NMS”

    symbol.results(:to_hash).output
    #=> [{:type=>”Stock”, :symbol=>”CFYH.PK”, :last_trade=>”10.49″, :industry_category=>””, :exchange=>”PNK”, :name=>”CITIGROUPFDGYAHOO”}, {:type=>”Stock”, :symbol=>”YA-U.TI”, :last_trade=>”12.75″, :industry_category=>””, :exchange=>”TLO”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.HM”, :last_trade=>”12.50″, :industry_category=>””, :exchange=>”HAM”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.BE”, :last_trade=>”12.51″, :industry_category=>””, :exchange=>”BER”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.DE”, :last_trade=>”12.50″, :industry_category=>””, :exchange=>”GER”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.DU”, :last_trade=>”12.48″, :industry_category=>””, :exchange=>”DUS”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.F”, :last_trade=>”12.29″, :industry_category=>””, :exchange=>”FRA”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.MU”, :last_trade=>”12.93″, :industry_category=>””, :exchange=>”MUN”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHO.SG”, :last_trade=>”12.51″, :industry_category=>””, :exchange=>”STU”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHOO.MX”, :last_trade=>”210.00″, :industry_category=>””, :exchange=>”MEX”, :name=>”YAHOO”}, {:type=>”Stock”, :symbol=>”YHOO.BA”, :last_trade=>”7.50″, :industry_category=>””, :exchange=>”BUE”, :name=>”YAHOOINC.”}, {:type=>”Stock”, :symbol=>”YOJ.DE”, :last_trade=>”250.00″, :industry_category=>””, :exchange=>”GER”, :name=>”YAHOOJAPAN”}, {:type=>”Stock”, :symbol=>”YOJ.F”, :last_trade=>”304.60″, :industry_category=>””, :exchange=>”FRA”, :name=>”YAHOOJAPAN”}, {:type=>”Stock”, :symbol=>”YOJ.BE”, :last_trade=>”216.11″, :industry_category=>””, :exchange=>”BER”, :name=>”YAHOOJAPAN”}, {:type=>”Stock”, :symbol=>”YAHOF.PK”, :last_trade=>”318.58″, :industry_category=>””, :exchange=>”PNK”, :name=>”YAHOOJAPANCORP”}, {:type=>”Stock”, :symbol=>”YAHOY.PK”, :last_trade=>”121.75″, :industry_category=>””, :exchange=>”PNK”, :name=>”YAHOOJAPANCORP”}, {:type=>”Stock”, :symbol=>”YHOO”, :last_trade=>”15.29″, :industry_category=>”InternetInformationProviders”, :exchange=>”NMS”, :name=>”Yahoo!Inc.”}]

  21. Erik Walters says:
    Reply

    Great Gem. Thank you for writing it. I don’t know if you still monitor this blog, but I will take a shot anyway….

    When you send in multiple symbols to the quote interfaces, does the api send them to yahoo on at a time or in a batch?

    If they are sent in batch, do you know the most symbols you sent?

  23. Erik Walters says:
    Reply

    I don’t know if you work on bugs for this gem anymore. But I found an interesting one. The symbol: “LO” returns “LO” symbol in the ticker_trend and not the actual symbol field. I have it working for all the others and I checked that it is a valid symbol in yahoo.

    symbol=YahooStock::Quote.new(:stock_symbols=>[‘LO’])
    symbol.standard
    symbol.results(:to_hash).output

    • Erik Walters says:
      Reply

      I take this issue back. It seems to be something, where if I send them in batch, some of the symbols get placed in a different field.

  27. Pingback: Updated Site | An Autonomous Agent

Leave a comment