Speedy data

Speed is an important part of my bot development. When it comes to storing data, the options (data structures) I’ve been working with are array, collection and dictionary. When I Google for articles on speed, I get lots of information pointing to dictionaries as being the fastest for programming. But the main point of interest in the articles is the speed of looking up data. To find something in an array, all elements have to be looped through until it’s found (or isn’t). The use of keys in collections and dictionaries makes lookup faster as they can be targeted without looping, as long as you know the key(s). There are other advantages that make dictionaries preferred to collections, however they seem less important when looking at speed. A disadvantage of arrays comes when changing its size. Collections and dictionaries can have elements added without problems. To do the same with an array, we have to change its size to accommodate more data, which involves more time as the original is put into temporary memory and a new, re-sized, array created with the existing data then added to it.

This leaves me thinking that the two main speed disadvantages of arrays is searching for data and resizing. Here’s the question I’m looking to answer – if I know the size I want the array to be and I know the location of all the data held in it (so I can refer to each element directly) is there a speed benefit to using collections or dictionaries? Any help appreciated.

Advertisements

6 thoughts on “Speedy data”

  1. Where do you host your programs? Cloud? You streaming? I think I/O and network latency combined with serialization are likely to be your biggest bottle necks. With python you can use cprofile to see what is slowing you down not sure what the VBA equivalent is.

    Like

    1. VBA and Visual Basic are two different entities. Trader247 is using Visual Basic to create his bot.

      He poses an interesting question the answer to which is somewhere at the back of my mind and will need coaxing out over the weekend.

      A few questions. Why do you want to use an array? I think you have answered the speed benefits yourself but I can clarify.

      BPT – http://www.betfairprotrader.co.uk

      Like

  2. >>> To find something in an array, all elements have to be looped through until it’s found (or isn’t).

    You probably know this anyway, but…

    If you use a List object to hold your collection of objects/things, the .Find() method of this class returns the first occurrence of the object or item you’re searching for – I haven’t tested it for performance but I think it will be faster than searching through every element of an array. There are other similar methods along the same lines as well (eg. FindAll, etc.).

    It also has a BinarySearch method you can use on sorted lists.

    Plus you can serialise/persist it very easily as well. I use it all of the time.

    I use C#, but here’s a link to the VB page from MSDN (VB looks very verbose and noisy compared to C/C++/C#/Java/etc…) 🙂

    https://msdn.microsoft.com/en-us/library/x0b5b5bc(v=vs.110).aspx?cs-save-lang=1&cs-lang=vb#code-snippet-1

    Cheers,

    Ken

    Like

  3. As a professional programmer I can tell you this: lookup times in a dictionary _can_ be faster than in an array, but this doesn’t necessarily have to be so.. On the flip side, iteration over an array _will_ certainly be much faster than over linked-type collections like a linked-list or a dictionary.

    However, having this in mind, as another commentator said, if you’re really concerned with speeding up your program, you should probably be looking for other things too. Algorithmic complexity and data structures are indeed something to consider, but since you seem to come from a VB background, there are other things equally, if not more, important..

    VB is inherently slow compared to many other languages, and also things such as network latency play a huge role.

    I would advice you look into a reliable method of profiling for your language, since there’s nothing worse than trying to optimize something based on hunches, without cold hard data to point you to where the real gains can be. That’s just plain wasting your valuable time.

    https://en.wikipedia.org/wiki/Profiling_(computer_programming)
    There are probably many tools to aid in doing this, but this can be done manually in every language usually using the language itself. I recommend trying to keep everything as simple as possible, but be very methodical about it.

    Also, it wouldn’t hurt to consider switching to a faster language such as python (still not super fast, but much so than VB, very very powerful and easy-ish to learn), or C# (very fast, without being hardcore like C++), but as I said, it may very well be that you have much bigger bottlenecks elsewhere.. until you don’t profile you won’t know.

    Like

  4. Many valid points from n00bmind here, but I do disagree that python is usually faster than VB. Python is an interpreted language, while VB is a compiled language. Not going into details here, but this means that for most cases VB will outperform python in terms of performance.

    But as you say, handling network latency properly and doing performance profiling is much more important than choosing between arrays and lists. The latter would probably be an example of premature optimization.

    Like

    1. If we’re talking about VB.NET here, then yes you’re right, it will probably be faster than python, although there’s also Jython and other interesting techniques (check https://www.youtube.com/watch?v=Iw9-GckD-gQ).
      Anyway, in my view python has many advantages over any VB dialect that go beyond speed only, so I still think it’s a better option in almost any situation.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s