Tag Archives: automated trading

Updated updating and simply greening

First, Liam replied to July ’17

You can set the conflation during the market subscribe request, defaults to 0 but you can choose how often you want to receive it. It’s odd because it’s a handy feature but none of the trading apps seem to use it.

I know betfair use a piece of open source software called Kafka for managing streaming updates but there is probably something else handling slow consumers (conflation). Sockets are complex but at a high level it is my understanding that once the receivers buffer is full the server is told this which then allows the server to halt sending more data. I believe it is then a case of updating that ‘halted’ data to prevent the consumer receiving ‘stale’ data.

Hopefully that makes more sense.

 

Thanks Liam, it does. I appreciate this is more in-depth stuff and beyond most of us but I am interested in these things. It’s the sort of detail that gets exploited by those in the know before anyone else cottons on. I wonder if the lack of utilisation by trading apps is because setting it for optimal performance would be very specific to an individual socket, the machine’s processor, what else it was doing at the time and the level of market activity?

test_passed_stamp

Now, bench testing complete, I’ve put the new Oscar live on the Aus horses only (staged implementation, sounds good). As mentioned in the last post, there are two areas where significant changes have been made to the code.

I trialled two different methods of navigating the markets. One is quite basic and simple code, the other more complex but I thought it would handle any troublesome delays better. It was over complicated though and as I worked through the errors whilst testing, it became bloated and confusing. Impact wise, the simpler code is only one extra step on an if-then-else statement. And it’s the same amount of steps as the old code. I’ve set it to update after a set time, 1:45am, then select first race in list shortly after. I use the day number, as integer, to check if the process has completed today. Previously the list update was performed within a set time period, which was fine when there were regular updates at the desired refresh rate. But with streaming, if there’s a market loaded that’s a few day’s off, as is sometimes the case for big events, there are no regular refreshes. This can mean that the set time window see’s no updates to run the list refresh code.

That change is almost not needed because of the next change. I’ve coded to turn full streaming off when not within the trading time zone. This forces regular refreshes. The above change will still come into use though if there are any issues causing a delay (can’t think what, but the code is better now, I’m sure).

Finally for this update, I’ve employed the green up trigger to replace my code. The module is less than a quarter of it’s previous size and the trigger takes care of cancelling any unmatched bets, submitting a greening trade and then chasing every second until matched. Splendid.

Advertisements

July ’17

Liam commented on Streaming and NSW

Its important to note that latency from streaming is a function of how quickly you can process the updates. If your CPU is unable to process the updates quick enough betfair will apply conflation which is what you are probably seeing. Of course network latency can impact this but its is more likely to be due to CPU / algorithm you are using to process the stream.

“””
Writes to your connection are directly effected by how quickly you consume data & clear your socket’s buffer
Consuming data slowly is effectively identical to setting conflation.
If you receive conf=true flag on a market – then you are consuming data slower than the rate of deliver
“””

Thanks for the comment.

This is something new to me and trying to find further info doesn’t reveal much. I haven’t seen any reference to it in the Gruss error log. Although I’m not entirely sure I understand this correctly. The statement in the quotes talks of “effectively setting conflation” by consuming data slowly. But who sets conflation? Which I guess to mean that updates are delivered in groups, merged, rather than individually. Is it something you can do from this side? Or is it a decision made by Betfair? But how would they know how you’re processing the data? My understanding of streaming is that it arrives at your computer as is, stored in a local cache(?), then processed by the program/bot. Would it be delivered/streamed to the computer conflated? Or conflated in the local cache for processing? I don’t know but it sounds interesting and any more information would be appreciated.

I’m only using Gruss at the moment as I’m not really excited by the prospect of coding for streaming in VB. My ventures into VB continue but not with Betfair. I finished a basic lay stake calculator and am now on with a more advanced version, trying to get a rolling list to display neatly. Also tested some scraping code and played with excel controls. I find Visual Studio so much more advanced than the VBA IDE that coding/debugging is a lot clearer and faster.

Back to trading. I’ve two test bots in coding at the moment. One is a reworking of a bot I’ve trialled on and off. Simply, it scrapes data and then trades. The thing is, each time I go back to it I have to change a chunk of code as the websites, as is always the case, alter their layout or swap some data from html to a type that doesn’t scrape easily (javasript/flash?). On the bright side it does mean that I learn some new code or technique.

The second test bot is an update to Oscar to take in the recent changes introduced with streaming. There is two parts to this – market navigation and greening. I’m having problems with the bot stopping on suspended/closed/abandoned events and have missed a number of days across the different bots. Although some change was made by the Gruss team, improving the situation, it is still happening. So to help from my end, I’m changing the way my bots refresh the markets (list of available events for each bot) which are held in the Quick Pick List. This will also include a change to how the bots jump from one market to the next; switching off full stream, which forces a specific refresh rate, prior to leaving an event, then switching full stream on once a new market is loaded and certain conditions are met.

EDIT: Team Gruss have said they’re on with fixing this problem 🙂

A new feature now available is the eagerly awaited and much anticipated greening trigger (it really has been wanted for ages and requested loads on the forum). My greening code is quite messy with multiple attempts to green. It’s been a crude hack from the early day’s that I’ve bodged to work rather than start fresh and I’m happy to see it go to be replaced by a simpler solution.

Oscar now and the UK dogs. Hmm. It is positive, that’s about it.

chart_ukdogs170731

Aus horses. Also positive…

chart_aus_horse170731

Aus dogs. The story of the 6p I wish I still had. Seriously though, there just isn’t the liquidity so the result could be anywhere to be fair. I stopped this bot at the beginning of August with a profit of 12p since May. I don’t see any reason to risk a big loss on these markets where it is unlikely to ever make it back.

chart_aus_dogs170731

Us horses have produced again with 0.121% for July. June was 0.118%. Not vastly more markets traded than the Aus dogs but there are races that get as much money as the bigger UK dogs races and the higher liquidity gives more opportunities to trade. I guess these races are the televised ones?

chart_ushorse170731

I’ve been invited along to the Ebor festival at York races on the Saturday. I’m taking a bit more out of the bank for spend. It’ll be the first big race I’ve been to having only been to the races three times before – regular events at Doncaster, Ripon and Pontefract – so am looking forward to it.

June ’17

The big event of June was the release of a beta streaming version of Gruss. As it is in development, the Gruss guys requested feedback and there were a number of bugs highlighted by users. We are now on the sixth release, which is good to know the program is being worked on and improved. The bug that I found related to moving on from suspended markets but after I fed back via the forum, a fix was quickly released. It took a little time for me to grasp the full effects of streaming. At first I thought the refresh was poor as updates were very random. The stream only updates when there is something to update, ie market activity, and doesn’t waste bandwidth by refreshing the same data repeatedly, as before streaming. This gives a refresh chart that can have quite large gaps between updates, especially on markets with some time to go before off. Now, with only a few minutes to off, the refreshes come in more than every 200ms. As there are no requests for price data, there’s no added delay. To note, the charts on the VPS are showing lower times than those at home. The request delay will still be relevant when placing orders in the market but no data is available on what it is.

Another major event, for me, was the changes to the Aus turnover eligibility. I posted about this here. I’m on with the coding around this. I have a section of code that only runs once when a market is selected and then isn’t run every refresh. I’m adding the NSW code there, which was straight forward for checking against the list of courses but tracking the traded back bets over a week is a little more complicated.

The UK dogs have done good this month. I’m considering adjusting the stake range to allow higher bets. I want to trial it on specific markets first. I’m thinking of those that are televised and tend to have much higher activity.

chart_ukdogs170630

An improved chart from the Aus horses compared to recent months. It’s nice to see a good return as I was beginning to lose patience with it and was considering stopping this bot. It’s been a long time since I ran Oscar on the UK horses as there was no value in it for me and I was thinking the Aus horses were going the same way. They still might, to be fair. But for now, with this, it will continue.

chart_aus_horse170630

US horses – better result than not trading them at all. There are some really well funded races that I’m missing, purely down to start times been well off. I’ll continue moaning about this point until I finally get a solution in place (I’ve had some good suggestions from you but the code don’t write itself, I should get on with it).

chart_ushorse170630

Definitely lower activity on the dish-lickers. An unfortunate loss keeping the return in the negative. Not much harm in continuing for now, I consider this my experimental contribution (why not?)

chart_aus_dogs170630

A comment, an icon, rounding and an offer.

US Horse Racing Off Times have been a problem for bot developers since the year dot. The US off times are just a guide and are not religiously adhered to like the rest of the world. Initially I got around this by polling the Time to Post stat that gives a guide of when the off time is due. This value can be scraped from a number of different sites. Even this was a little hit and miss. What I eventually landed on was waiting till the overround was less than 105% to indicate that the race was about to go off. Works quite well in sparse US markets where all the money comes in at the end before the off.

Thanks for this comment. I’ve looked into scraping times and have found a few sites. Not made any attempts to integrate yet. The idea of watching the overround is very good and something I hadn’t thought of. This goes on the to-do list, thanks.

 

 

I’ve been coding an app in Visual Studio to replicate a spreadsheet I’d made when I was doing the matched betting thing. I added an icon and thought I’d add it to this site, so if you haven’t noticed, it should be at the top of the address bar and look like this –

favicon3

I like the simple look, which is fortunate. And it should show up as different to WordPress in any bookmarks, or in the blog roll on green-all-over (It’s been quiet over there recently).

 

The app I’ve made calculates the lay stake at given odds. A problem I encountered whilst testing my little app was a difference between what I calculated and what I was seeing on the Betfair site. It’s only small but at large stakes it’s noticeable and any discrepancies are worrying, suggesting an error in the formula used. My error came down to rounding. Initially I was just rounding the result only to 2dp. Each step of the formula requires rounding to 2dp to keep it accurate. At no point can you place a lay stake of £50.083333. Attention to detail is key, my mistake.

 

Talking of attention to detail, I recieved an offer to try out a VPS in Dublin with less than 3ms delay. That’s extremely fast for sports trading and probably as good as it gets without co-location (something I’m sure goes on). I had looked at getting a Dublin based VPS when I was setting up but was not willing to pay the obvious premium rate. The offer I received came via Twitter and the first thing anyone does is look at the website of the people making the offer. I remember this company was looking for people to trial its VPS about a year ago. Their website still has some pages that haven’t been altered from the template settings – the blurb you get when you start a website from a pre-made layout offered by these site builder outfits. There’s a lot of trust when putting your bots onto a VPS and you want to know you’re in safe hands that pay attention to detail. I declined the offer.

May ’17 – and the art of separation.

Work on the VB bot was frustrating me so I decided to pause it and have a play with Oscar, my VBA bot. I’ve nearly always run one instance of Oscar, navigating between different markets and sports based on some preset criteria. I decided to split the sports, running an instance for UK dogs and one for Aus horsies. This has the benefit of not missing conflicting events across the two sports. The reason Oscar didn’t do this originally is because back in the day, Betfair charged for making excessive calls above a relatively low amount. This changed some time back but I hadn’t.

This new set-up runs well and so I added Aus dogs, also running in its own instance. And why not US horses? OK, they are now covered in another instance. (Previous attempts at US horses had not seen many trades but most races were missed in favour of the other markets.)

I’ve monitored some cross-over times on the VPS and I haven’t seen any drop in performance. At some points in the morning, three of the bots are running at 0.2s refresh rate but I’m still getting a delay of less than 20ms on each.

I did notice the other day that the US horses were buggered by some error in the stated off time. For one venue the times were in the quick pick list but when the markets were selected, the off time was around an hour and a half out. This may have been an API issue but if I see it again I’ll look at coding to handle the mismatch between the two.

Next, Chris commented on Speedy data 2  –

Very interesting articles about bot speeds. I have been looking into the same. I have looked at my algorithms and have improved them. They now return values within 3-4 ms. However the main bottle neck is the price refreshes. Without streaming they currently have a price refresh at 200ms, my prices can be 180 ms out of date. If I was able to implement streaming I could improve my robots speed by a huge amount (probably 100ms), dwarfing any gains that could be made over optimising my robots. So I would suggest that the bottleneck is in your price refreshing and you could see a large improvement with your bots if you were able to stream prices.

Thanks for the comment. 3-4ms is fast and I haven’t seen those speeds from my bots yet. How are you timing the code? And what language are you coding in? The arrival of streaming made me less eager to push on with the VB bot as I don’t want to put all the time in to get the code perfect just to see it become old-hat overnight. Gruss, the software I use for my VBA bots, are releasing a beta streaming version soon. When I’ve had a go with that, I’ll look at how to stream with VB. There’s no point at all in not streaming if it’s faster and as reliable (collective eye roll) as the API-NG.

And now, some charts.

UK dogs have done ok, nice steady performance.

chart_ukdogs170531

Aus horses continue to throw some bad results. The three sharp drops in this chart have different causes. The first is actually 3 losing markets together, so no problem there. The second was an error, a problem I haven’t seen for a while where an extra lay is submitted for some reason. I’ve previously thought this is down to timing and the bot missing signals at specific points, eg when greening occurs and a bet is taken at the same time as a CANCEL-ALL command is triggered. The third was just a bad run of multiple bets being placed within the stoploss window, all eventually losing trades. When I’ve attempted to overcome this particular event in the past, the number of trades significantly reduced. I may look at this again, specifically in the Aus horse markets, but with a more complex solution.

chart_aus_horse170531

NEW – Aus dogs, although not many markets, has a good looking chart, certainly one to watch. Stakes are still hovering around £2 for now.

chart_aus_dogs170531

NEW – US horses, only 3 days here so wait and see what happens in June. Interesting to see average bets per market at 9.9, with other sports being

UK dogs = 5.6

Aus horses = 6.1

Aus dogs = 2.9

chart_ushorse170531

You can find all of Oscar’s UK dogs charts on a single page now – see here.

 

Australian greyhounds and other activities

As mentioned in a previous post, an Oscar clone has been let out of the traps (ha ha, out of the traps – greyhounds, traps, letting out of the… yeah, yeah) and has been trading the Australian greyhounds for the past week. The liquidity is generally lower on these markets, especially for the earlier races but some of the later races are ok. With settings almost identical to those for the UK, only 27 races saw any trading with a total 115 bets settled. Profit of 27 pence (coincidentally) and £270.69 traded, giving a P&L/TV of 0.1%, which is not an unusual figure for Oscar. I’ll let it run for now to see what results come in.

aus_dogs_170521

Other activities

I know many of you will have it on your mind but for those that are new here – back in October I did a post on stakes ending with me wondering what to do with any profits. I gave four possibilities, as I saw it –

  1. Leave it where it is, doing nothing.
  2. Create a second Betfair account, for other/future bots or split activities.
  3. Remove from Betfair, put it in savings to be returned to Betfair when required.
  4. Remove from Betfair, spend it (I doubt this will happen).

Option 2 is not an option. I thought option 3 would be the one to go with but the wife suggested (after pointing out that there wasn’t vast amounts to play with ” and after all that time you spend on the computer“) that I should buy something – option 4 – which was MY LEAST favoured option. So after thinking it over and realising that she was right, because that’s how it is, I decided to part company with my faithful 20+ year old mountain bike (Claud Butler frame swapped for a scrap ’85 Ford Escort, crank and gears from a previous GT, all other parts swapped/added separately pre 2000, except tyres) and purchase a brand spanking shiny new modern lighter full suspension ally framed disk braked, mountain bike. I’ve been using it regularly for the past few months to try (really try) to improve my fitness. This is most certainly a work in progress.

img_20170114_140522068
Old
img_20170114_140422315
New (when it was new)

(p.s. I still have the old bike, just can’t bring myself to drop it at the tip after all these years)

 

Speedy data 2

My Speedy data post generated a few comments and some discussion. I really appreciate people taking the time to get involved and share their knowledge and views.

The first comments came via twitter from TraderBot (here and here) with a link to stackoverflow. This is a site I’ve found to be really useful to get help with many programming issues in multiple languages (That reminds me, I keep meaning to do a list of what I use – apps and sites). The Q&As linked to, although relating to Java, are an interesting read with an answer to the speed question, in summary, of “it depends on what exactly you want to do/measure”.

LiamPauling commented on the post asking where I’m hosted and do I stream? I’m cloud hosted and not streaming. He continues that he thinks bottlenecks are more likely elsewhere, which, after further reference in later comments, seems to be a good point.

Betfair Pro Trader asked why I wanted to use an array. It’s not that I want to use an array more than any other data structure, I was looking at getting the best solution, if such a thing exists (which is becoming less clear).

Tony, via Twitter, suggested running a test code with the different structures used. This could be useful but I was put off from this initially by the confusion I was getting from reading differing opinions based on various implementations of arrays, collections and dictionaries (and later, lists). At this point I was thinking that the optimum stucture is dependant on the specific use and there isn’t an exact answer to my speed question.

Next, a comment from Ken. He points to Lists as it’s something that he uses regularly and he talks of some of the benefits. Again, I’d previously come across articles saying lists were slow but maybe I was too quick to dismiss them. Betfair Pro Trader has also suggested using lists and dictionaries combined. Ken adds that he codes in C# (C sharp) but I think for the purpose of data structures and speed they are similar (they, C# and VB.net compile to the same language and run against the same runtime libraries).

n00bmind added a detailed comment. He makes the point that the advantages of one structure over another are not always so, as mentioned above. Also, he goes on to agree with previous comments that my speed question may be missing the main issues – those being the program/algorithm itself and network latency. Further advice is given about profiling (something, as a specific process, I haven’t come across before) and maybe using a different language, such as Python (I have only a basic understanding of Python from messing with it on my Raspberry Pi).

Finally, Jptrader commented, agreeing mostly with n00bmind, and others, about looking at “handling network latency properly and doing performance profiling”.

Although a simple answer hasn’t been found (because there isn’t one), I’m guided by these comments to focus more on my code, handling serialization and latency, making the algorithm efficient and using the data structures that work for now, whether that’s arrays, collections, dictionaries, lists or a combination of. Moving to another language just isn’t feasible for me at the moment, it’s taken me over a year to get a running bot in VB, with limited hobby time. I am happy to accept that another language may have it’s advantages, so would advise others to look at this for optimising their bots performance (for me the advantage will be seen moving from VBA to VB.net).

The testing I’ve done hasn’t shown any particular advantage of the different structures. From my searches on the web I think this could be due to the relatively small amount of data I’m handling (many articles talk of data lines in the 10s to 100s of thousands when comparing structures). An error on my part also had me making double calls for data with my bot which added to my difficulties and questions initially.

I have plenty to be getting on with for now and will continue looking to improve my bots. Thanks again for all the comments.