Not finding errors

I read this while looking for help –

“There’s only two types of programming languages: the ones people complain about, and the ones no one uses.”

I’d like to think it’s true.

I was searching for help on handling error 9 which pops up when referring to an array that isn’t initialised. I was seeing this error when monitoring the MyBets sheet. My code for logging the changes to MyBets works like this –

• On change to MyBets – lift the entire data range into an array (Array1)
• Check this data against previous data lift (Array2) to find any changes
• Log changes to another array (Array3)
• Once Array1 is checked, place Array3 in another sheet
• Save Array1 to Array2 for next time
• Repeat
• On market change – erase Array2

This doesn’t work on the first change to MyBets on a new market as Array2 isn’t initialised. It could be initialised if it was a known size but this can change dependant on other data I add to Array1. I could write code to size the array on each market entry, but I was looking for a quick way of saying – if referencing the array gives an error, load it with the current data and move on. I found a way to do this using the Not operator. The code looks like this –

If Not Not Array2 = 0 Then…

Now, instead of an error when referencing Array2, this “Not Not” will return a 1 or 0, so the code flows and continues unhindered.

Reading around various forums it’s said that this method is actually a bug from VB6, which is what VBA is based on. And some say that this method may disappear if the bug is fixed. I also read that VB6 is no longer maintained, so the chance of the bug being fixed is low.

For me, this works for now so I’ll use it where I need to.

What’s going on?

[I started this post last week and now I think I’ve solved the problem, but as I’ve written it, you can read it. solution at bottom.]

Over the past two weeks, Oscar, my UK dogs bot, has had six markets failing to trade out or green up. After the first two, I stopped for two days whilst I got round to checking the code but everything was fine, as it should be as I hadn’t changed anything. I restarted and after another few days of trading ok, it happened again, so it’s now stopped. I can’t see anything particular about the markets it’s happening on, and the connection from my VPS appears good. I’m now thinking of adding some extra code to record more details in hope of catching what’s going on.

I’m less narked than you may think as the result so far is I’m up £40 from the errors. But obviously I can’t let it carry on.

In other news, I’ve been reading. A challenging task for someone who flits from one thing to another. Anyway, this had lead to some really good coding sessions (from my point of view) but nothing live yet, only recording and paper trading.

I put quite a bit of time into code that aims to measure the chance of being matched after a trigger is reached and over what period that match would take place. Recording price movement is pretty easy and amount matched over a short period in one direction is also simple. But tracking price, piq and amount matched if the price moves against and then returns, then goes positive, before returning against, is a job for good code. (I’ve not had it marked yet, so no conclusions.)

And I’ve just put on test some code for showing which runner has won on inplay markets. My aim is to know the winner straight away and not have to have a separate process. I have some old code that would take my bets sheet (paper trades) and check it against the RP results page (by runner) after all the races were finished but it requires random fettling to keep up with the website changes. I haven’t used it for some time. It also only happened once a day. My new code works with obvious winners, reaching and staying at 1.01, but then handles close calls individually, with unknowns highlighted for my attention at the moment (this still requires input from me but I’m not checking through every market). I get a rolling update from this.

Inplay is not something I’ve botted with any interest before. But a post on the Gruss forum caught my attention. It regards a simple strategy shown in a YouTube video. It got my interest in the inplay side of things. Rather than throwing bets in and seeing what happens, I thought I’d approach this with a plan and code my way to results rather than bet my way there. If the testing comes to nothing, no waste, I’ve got some cracking reusable code.

[SOLVED, I think]

As mentioned at the top, I think I’ve found the problem.

It’s a time issue. My server clock was 33.7 seconds behind. You can find this out by going to the website – time.is. (My laptop is 0.8s out.) UK dogs are quite sharp with suspending, 30secs+ could easily miss my greening period.

A simple fix? No. Why would it be? Multiple tries at pointing my desktop clock to various internet ntp clocks resulted in a 0.1 second increase in error (huh). I Googled around and found an article on Dimension 4. Downloaded it. This took a few attempts with different servers and then bingo! Back to time.is for an exact result. Bot restarted. Time will tell (time, it’s a time pun, ha ha).

 

Updated updating and simply greening

First, Liam replied to July ’17

You can set the conflation during the market subscribe request, defaults to 0 but you can choose how often you want to receive it. It’s odd because it’s a handy feature but none of the trading apps seem to use it.

I know betfair use a piece of open source software called Kafka for managing streaming updates but there is probably something else handling slow consumers (conflation). Sockets are complex but at a high level it is my understanding that once the receivers buffer is full the server is told this which then allows the server to halt sending more data. I believe it is then a case of updating that ‘halted’ data to prevent the consumer receiving ‘stale’ data.

Hopefully that makes more sense.

 

Thanks Liam, it does. I appreciate this is more in-depth stuff and beyond most of us but I am interested in these things. It’s the sort of detail that gets exploited by those in the know before anyone else cottons on. I wonder if the lack of utilisation by trading apps is because setting it for optimal performance would be very specific to an individual socket, the machine’s processor, what else it was doing at the time and the level of market activity?

Now, bench testing complete, I’ve put the new Oscar live on the Aus horses only (staged implementation, sounds good). As mentioned in the last post, there are two areas where significant changes have been made to the code.

I trialled two different methods of navigating the markets. One is quite basic and simple code, the other more complex but I thought it would handle any troublesome delays better. It was over complicated though and as I worked through the errors whilst testing, it became bloated and confusing. Impact wise, the simpler code is only one extra step on an if-then-else statement. And it’s the same amount of steps as the old code. I’ve set it to update after a set time, 1:45am, then select first race in list shortly after. I use the day number, as integer, to check if the process has completed today. Previously the list update was performed within a set time period, which was fine when there were regular updates at the desired refresh rate. But with streaming, if there’s a market loaded that’s a few day’s off, as is sometimes the case for big events, there are no regular refreshes. This can mean that the set time window see’s no updates to run the list refresh code.

That change is almost not needed because of the next change. I’ve coded to turn full streaming off when not within the trading time zone. This forces regular refreshes. The above change will still come into use though if there are any issues causing a delay (can’t think what, but the code is better now, I’m sure).

Finally for this update, I’ve employed the green up trigger to replace my code. The module is less than a quarter of it’s previous size and the trigger takes care of cancelling any unmatched bets, submitting a greening trade and then chasing every second until matched. Splendid.

A comment, an icon, rounding and an offer.

Drive Blind commented –

US Horse Racing Off Times have been a problem for bot developers since the year dot. The US off times are just a guide and are not religiously adhered to like the rest of the world. Initially I got around this by polling the Time to Post stat that gives a guide of when the off time is due. This value can be scraped from a number of different sites. Even this was a little hit and miss. What I eventually landed on was waiting till the overround was less than 105% to indicate that the race was about to go off. Works quite well in sparse US markets where all the money comes in at the end before the off.

Thanks for this comment. I’ve looked into scraping times and have found a few sites. Not made any attempts to integrate yet. The idea of watching the overround is very good and something I hadn’t thought of. This goes on the to-do list, thanks.

 

 

I’ve been coding an app in Visual Studio to replicate a spreadsheet I’d made when I was doing the matched betting thing. I added an icon and thought I’d add it to this site, so if you haven’t noticed, it should be at the top of the address bar and look like this –

I like the simple look, which is fortunate. And it should show up as different to WordPress in any bookmarks, or in the blog roll on green-all-over (It’s been quiet over there recently).

 

The app I’ve made calculates the lay stake at given odds. A problem I encountered whilst testing my little app was a difference between what I calculated and what I was seeing on the Betfair site. It’s only small but at large stakes it’s noticeable and any discrepancies are worrying, suggesting an error in the formula used. My error came down to rounding. Initially I was just rounding the result only to 2dp. Each step of the formula requires rounding to 2dp to keep it accurate. At no point can you place a lay stake of £50.083333. Attention to detail is key, my mistake.

 

Talking of attention to detail, I recieved an offer to try out a VPS in Dublin with less than 3ms delay. That’s extremely fast for sports trading and probably as good as it gets without co-location (something I’m sure goes on). I had looked at getting a Dublin based VPS when I was setting up but was not willing to pay the obvious premium rate. The offer I received came via Twitter and the first thing anyone does is look at the website of the people making the offer. I remember this company was looking for people to trial its VPS about a year ago. Their website still has some pages that haven’t been altered from the template settings – the blurb you get when you start a website from a pre-made layout offered by these site builder outfits. There’s a lot of trust when putting your bots onto a VPS and you want to know you’re in safe hands that pay attention to detail. I declined the offer.

Speedy data 2

My Speedy data post generated a few comments and some discussion. I really appreciate people taking the time to get involved and share their knowledge and views.

The first comments came via twitter from TraderBot (here and here) with a link to stackoverflow. This is a site I’ve found to be really useful to get help with many programming issues in multiple languages (That reminds me, I keep meaning to do a list of what I use – apps and sites). The Q&As linked to, although relating to Java, are an interesting read with an answer to the speed question, in summary, of “it depends on what exactly you want to do/measure”.

LiamPauling commented on the post asking where I’m hosted and do I stream? I’m cloud hosted and not streaming. He continues that he thinks bottlenecks are more likely elsewhere, which, after further reference in later comments, seems to be a good point.

Betfair Pro Trader asked why I wanted to use an array. It’s not that I want to use an array more than any other data structure, I was looking at getting the best solution, if such a thing exists (which is becoming less clear).

Tony, via Twitter, suggested running a test code with the different structures used. This could be useful but I was put off from this initially by the confusion I was getting from reading differing opinions based on various implementations of arrays, collections and dictionaries (and later, lists). At this point I was thinking that the optimum stucture is dependant on the specific use and there isn’t an exact answer to my speed question.

Next, a comment from Ken. He points to Lists as it’s something that he uses regularly and he talks of some of the benefits. Again, I’d previously come across articles saying lists were slow but maybe I was too quick to dismiss them. Betfair Pro Trader has also suggested using lists and dictionaries combined. Ken adds that he codes in C# (C sharp) but I think for the purpose of data structures and speed they are similar (they, C# and VB.net compile to the same language and run against the same runtime libraries).

n00bmind added a detailed comment. He makes the point that the advantages of one structure over another are not always so, as mentioned above. Also, he goes on to agree with previous comments that my speed question may be missing the main issues – those being the program/algorithm itself and network latency. Further advice is given about profiling (something, as a specific process, I haven’t come across before) and maybe using a different language, such as Python (I have only a basic understanding of Python from messing with it on my Raspberry Pi).

Finally, Jptrader commented, agreeing mostly with n00bmind, and others, about looking at “handling network latency properly and doing performance profiling”.

Although a simple answer hasn’t been found (because there isn’t one), I’m guided by these comments to focus more on my code, handling serialization and latency, making the algorithm efficient and using the data structures that work for now, whether that’s arrays, collections, dictionaries, lists or a combination of. Moving to another language just isn’t feasible for me at the moment, it’s taken me over a year to get a running bot in VB, with limited hobby time. I am happy to accept that another language may have it’s advantages, so would advise others to look at this for optimising their bots performance (for me the advantage will be seen moving from VBA to VB.net).

The testing I’ve done hasn’t shown any particular advantage of the different structures. From my searches on the web I think this could be due to the relatively small amount of data I’m handling (many articles talk of data lines in the 10s to 100s of thousands when comparing structures). An error on my part also had me making double calls for data with my bot which added to my difficulties and questions initially.

I have plenty to be getting on with for now and will continue looking to improve my bots. Thanks again for all the comments.

A chart, I see

I’ll return to the speed question soon (as I have an idea for a testing bot, just needs writing), the results of the little testing I’ve done are not really good, as in they don’t really move me forward. Development of the VB bot is still progressing though, well my programming ability is, with the more I learn I add/change what I want to do but it’s all good. I’ve been playing with what data to collect/monitor and how to handle it (to avoid unnecessary bloating of the bot). As I like to visualise things, I’ve been presenting data in different ways. Below is one of the charts I created, just with excel, to show matched volumes. It covers the second favourite for the final two minutes before the off in a middle-of-the-road greyhounds race. The data is at 1 second intervals with a 4% decay added to project but not obscure the matching. The price can be seen to rise as it approaches off time (front of image) with last-price-traded at 4.0. It isn’t much use on its own but I like how it looks. Adding another parameter, or two, gives more meaning.

Speedy data

Speed is an important part of my bot development. When it comes to storing data, the options (data structures) I’ve been working with are array, collection and dictionary. When I Google for articles on speed, I get lots of information pointing to dictionaries as being the fastest for programming. But the main point of interest in the articles is the speed of looking up data. To find something in an array, all elements have to be looped through until it’s found (or isn’t). The use of keys in collections and dictionaries makes lookup faster as they can be targeted without looping, as long as you know the key(s). There are other advantages that make dictionaries preferred to collections, however they seem less important when looking at speed. A disadvantage of arrays comes when changing its size. Collections and dictionaries can have elements added without problems. To do the same with an array, we have to change its size to accommodate more data, which involves more time as the original is put into temporary memory and a new, re-sized, array created with the existing data then added to it.

This leaves me thinking that the two main speed disadvantages of arrays is searching for data and resizing. Here’s the question I’m looking to answer – if I know the size I want the array to be and I know the location of all the data held in it (so I can refer to each element directly) is there a speed benefit to using collections or dictionaries? Any help appreciated.

How’s it going?

Automated_trader commented –

“How’s coding your own bot going? I got a copy of Programming for Betfair today so gonna try and code my own too. I’ve no coding experience other than a little VBA for my current bots that run on Gruss so hoping it won’t be too big of a leap into the unknown. Might even start a blog to document my progress too.”

The coding is going really well with a bot now in testing. I doubt I’d have started without Programming for Betfair as I thought the amount to learn wasn’t worth it. The book helps by giving you everything you need to start auto trading, as in it gives you the code to request data, place bets and handle the responses from Betfair. It also gives you the basic tools to trade with such as profit take, fill/kill and stoploss. Once you’ve worked through the book, it’s just a case of adapting what you’ve learnt to get what you want.

My currently running and previous bots are written in VBA and running through Gruss. Before I started them I hadn’t written in VBA but picked it up from forums and searches. I hadn’t done any VB.net prior to this but understanding it isn’t that much of a leap. It’s the same style of object programming and the terms and layout are familiar after using VBA.

It’s true that any errors (after the corrections on the associated website are completed) are down to typos. A couple of times I thought it wasn’t me and something must have changed making the code not work. But it was me, with one mistake being a missing “A” in an url string. That took hours to find and I’d looked at the offending line of code more than once. Another biggy was missing a whole line of code, again taking forever to find. Sometimes the error message isn’t that helpful, not to me anyway. This one was an “overload” which, I’ve since found out, means you’re trying to put more variables into an object than you have declared. That’s what the missing line was, a variable declaration.

Some things to note, firstly, Visual Studio (VS) uses IntelliSense which highlights errors as you go. To begin with, this is quite annoying as you’ll finish a module and there’s a list of errors, so you spend time looking for what you’ve done wrong – which is nothing. The errors disappear when you complete the next couple of pages in the book. I found it’s better to ignore most of them until you get to a point where the book tells you to try running the code. The ones to correct as you go are missing or expected characters, such as “(“ expected.

Second – VS includes an autocomplete function. This will really mash your head at some points as you try to add a new declaration and it changes it to something else. If this happens, a quick tap on backspace should swap it back to what you typed.

Third – I find that the first run of the code after starting VS can sometimes fail. Just stop and run again. This problem goes once you publish your project.

If you enjoy coding you shouldn’t have any difficulty with it, just need patience. Let me know how you get on.

Interesting test results

I’ve been recoding my horse racing form bot (Bot 3), trying to get it to work without errors. It scrapes a lot of data for each race – runner form, jockey form, track data, going, weather. Having got it to a point where it runs okay with me sat in front of it, I decided to run it live on the VPS but without placing bets, allowing performance and data collection checks. To keep things separate I’m running this on Betdaq with a slow refresh and, as I’ve not used it for a while, I’ve added a short section of code to place some pointless bets based on where the market odds are, nothing to do with the bots purpose but hopefully keeps the exchange happy.

The results so far are good and the bot is gathering most of the data without error. Any missing data isn’t causing the bot to stop, however, I’d like it to make multiple attempts to gather it, so some adjustments are needed. The algorithm for choosing a selection to back or lay is operational but requires refining. I’m basically calculating my own ‘betting forecast’ and backing or laying a selection depending on how far away from my suggested odds it is trading.

The interesting part comes from the short bet placement code I added. Of the 12 bets it placed over the five days testing, only 1 of the selections won. Obviously I’d backed them all.

 

My theory for this code was – if I place bets in efficient markets (which I thought close-to-off horse racing markets were), I will likely lose at a rate around my commission (5%) plus half the spread position. Therefore stakes of £0.1 are a small price for testing.

I know the small sample is hardly proof but with average odds of 3, none greater than 3.5 and results at 12, I’m going to keep this little code running to see how it pans out and update the blog on results. It will be interesting to find out if I just happened to drop into the strategy during a 10 long losing streak.

Oscar has returned

After a short holiday caused by this event, Oscar is now back at work, emotionless but committed.

Mike suggested using the Take-SP option to avoid missing trading out.

BPT commented he’d previously used Back at 1.01 and Lay at 1000 to catch the best available odds.

As a way of getting the bot going again, it was quite easy to replace the 2nd-next-best-price with 1.01 and 1000, so this is what I’ve done for now. This is only a temporary fix as the greening is still calculating one refresh behind.

As I improve the greening code, I will include the Take-SP as a last resort. I think this is better as, although the odds may be off, it is taken straight away without entering the market directly. With the code improved, the use of this should be less often anyway.