All posts by trader247

August ’17

The streaming version of Gruss is better now, none of the issues of late with missed market list updates or getting stuck on expired markets.

The new Oscar on the Aus horses has performed flawlessly (apart from another schoolboy error that lead to it trading one race the first day out and not moving on). I now need to add all those changes to the UK dogs, not that I have any trouble there but the code is neater and more efficient, so good practice to implement.

I haven’t done any more with the bots other than update Gruss when new beta versions are released, as I’ve been busy with other things.

A good chart from the UK dogs, less up and down than last month.

chart_ukdogs170831

Generally good for the Aus horses. The sharp up then down was a run of four races where only one back bet was placed. No offset or green bets placed/matched. I’m assuming there was either an API issue or connection problem. These races don’t seem to suffer from early starts, so it isn’t that. And they were different venues. One of those things.

The sharp drop prior to those four errors was a failed offset/greening sequence. There were 7 back bets placed/matched fully but the seventh lay bet wasn’t matched and then greening only partially matched.

These were all before the new code for greening went in, so I’ll have to wait and see if it happens again. I’ve probably said it before but in the long run, these errors tend to have very little effect.

chart_aus_horse170831

Although I praised the US horses last month, compared to the Aus dogs, this chart has pushed me to stopping the bot. Four missed offset bets resulting in a gain overall. I think the way my algo trades the markets, it just isn’t worth the risk with too few bets placed. There isn’t the turnover to recover in time for the next problem/error.

chart_ushorse170831

As mentioned last month I went to the Ebor festival at York for the Saturday races. It was a good event, nice weather, plenty to eat and drink. I’d gone with a tight limit on the betting and placed bets with the on-site bookies, to enjoy the experience. I don’t really like gambling though (in the traditional way), I can’t get past thinking I’m just going to lose as I don’t know anything about picking winners. Won nothing, surprise, but had loads of laughs and enjoyed getting to the front to see the horses race past the finish post, great atmosphere and very impressive. And Saturday night in York was a laugh too, finished with a kebab, “large… hic… everything on it mate… hic…  “, which seems like the best thing ever at the time but probably best forgetting about afterwards, rather than contemplating it’s content.

Advertisements

VPS data usage – streaming

Here is the data usage for my VPS, taken from the Tagadab dashboard.

vpsdata_use

As you can see, since streaming started in July, the amount of data/bandwidth used has reduced considerably. The differing previous monthly figures are down to various reasons – number of bots live/number of events/missed days due to errors/etc. June – 13GB – and July – 2.33GB – both had the four Oscars and one other running most of the time. Even when I wasn’t running full-stream, the data is still delivered via the stream, it’s just how the data is refreshed in Excel that is regulated, so it makes no difference to the bandwidth. (There are options to run off stream but I haven’t used these with the new version.)

The £0.00 value is because there is 50GB included in the package, charges are only incurred for exceeding this amount.

Updated updating and simply greening

First, Liam replied to July ’17

You can set the conflation during the market subscribe request, defaults to 0 but you can choose how often you want to receive it. It’s odd because it’s a handy feature but none of the trading apps seem to use it.

I know betfair use a piece of open source software called Kafka for managing streaming updates but there is probably something else handling slow consumers (conflation). Sockets are complex but at a high level it is my understanding that once the receivers buffer is full the server is told this which then allows the server to halt sending more data. I believe it is then a case of updating that ‘halted’ data to prevent the consumer receiving ‘stale’ data.

Hopefully that makes more sense.

 

Thanks Liam, it does. I appreciate this is more in-depth stuff and beyond most of us but I am interested in these things. It’s the sort of detail that gets exploited by those in the know before anyone else cottons on. I wonder if the lack of utilisation by trading apps is because setting it for optimal performance would be very specific to an individual socket, the machine’s processor, what else it was doing at the time and the level of market activity?

test_passed_stamp

Now, bench testing complete, I’ve put the new Oscar live on the Aus horses only (staged implementation, sounds good). As mentioned in the last post, there are two areas where significant changes have been made to the code.

I trialled two different methods of navigating the markets. One is quite basic and simple code, the other more complex but I thought it would handle any troublesome delays better. It was over complicated though and as I worked through the errors whilst testing, it became bloated and confusing. Impact wise, the simpler code is only one extra step on an if-then-else statement. And it’s the same amount of steps as the old code. I’ve set it to update after a set time, 1:45am, then select first race in list shortly after. I use the day number, as integer, to check if the process has completed today. Previously the list update was performed within a set time period, which was fine when there were regular updates at the desired refresh rate. But with streaming, if there’s a market loaded that’s a few day’s off, as is sometimes the case for big events, there are no regular refreshes. This can mean that the set time window see’s no updates to run the list refresh code.

That change is almost not needed because of the next change. I’ve coded to turn full streaming off when not within the trading time zone. This forces regular refreshes. The above change will still come into use though if there are any issues causing a delay (can’t think what, but the code is better now, I’m sure).

Finally for this update, I’ve employed the green up trigger to replace my code. The module is less than a quarter of it’s previous size and the trigger takes care of cancelling any unmatched bets, submitting a greening trade and then chasing every second until matched. Splendid.

July ’17

Liam commented on Streaming and NSW

Its important to note that latency from streaming is a function of how quickly you can process the updates. If your CPU is unable to process the updates quick enough betfair will apply conflation which is what you are probably seeing. Of course network latency can impact this but its is more likely to be due to CPU / algorithm you are using to process the stream.

“””
Writes to your connection are directly effected by how quickly you consume data & clear your socket’s buffer
Consuming data slowly is effectively identical to setting conflation.
If you receive conf=true flag on a market – then you are consuming data slower than the rate of deliver
“””

Thanks for the comment.

This is something new to me and trying to find further info doesn’t reveal much. I haven’t seen any reference to it in the Gruss error log. Although I’m not entirely sure I understand this correctly. The statement in the quotes talks of “effectively setting conflation” by consuming data slowly. But who sets conflation? Which I guess to mean that updates are delivered in groups, merged, rather than individually. Is it something you can do from this side? Or is it a decision made by Betfair? But how would they know how you’re processing the data? My understanding of streaming is that it arrives at your computer as is, stored in a local cache(?), then processed by the program/bot. Would it be delivered/streamed to the computer conflated? Or conflated in the local cache for processing? I don’t know but it sounds interesting and any more information would be appreciated.

I’m only using Gruss at the moment as I’m not really excited by the prospect of coding for streaming in VB. My ventures into VB continue but not with Betfair. I finished a basic lay stake calculator and am now on with a more advanced version, trying to get a rolling list to display neatly. Also tested some scraping code and played with excel controls. I find Visual Studio so much more advanced than the VBA IDE that coding/debugging is a lot clearer and faster.

Back to trading. I’ve two test bots in coding at the moment. One is a reworking of a bot I’ve trialled on and off. Simply, it scrapes data and then trades. The thing is, each time I go back to it I have to change a chunk of code as the websites, as is always the case, alter their layout or swap some data from html to a type that doesn’t scrape easily (javasript/flash?). On the bright side it does mean that I learn some new code or technique.

The second test bot is an update to Oscar to take in the recent changes introduced with streaming. There is two parts to this – market navigation and greening. I’m having problems with the bot stopping on suspended/closed/abandoned events and have missed a number of days across the different bots. Although some change was made by the Gruss team, improving the situation, it is still happening. So to help from my end, I’m changing the way my bots refresh the markets (list of available events for each bot) which are held in the Quick Pick List. This will also include a change to how the bots jump from one market to the next; switching off full stream, which forces a specific refresh rate, prior to leaving an event, then switching full stream on once a new market is loaded and certain conditions are met.

EDIT: Team Gruss have said they’re on with fixing this problem 🙂

A new feature now available is the eagerly awaited and much anticipated greening trigger (it really has been wanted for ages and requested loads on the forum). My greening code is quite messy with multiple attempts to green. It’s been a crude hack from the early day’s that I’ve bodged to work rather than start fresh and I’m happy to see it go to be replaced by a simpler solution.

Oscar now and the UK dogs. Hmm. It is positive, that’s about it.

chart_ukdogs170731

Aus horses. Also positive…

chart_aus_horse170731

Aus dogs. The story of the 6p I wish I still had. Seriously though, there just isn’t the liquidity so the result could be anywhere to be fair. I stopped this bot at the beginning of August with a profit of 12p since May. I don’t see any reason to risk a big loss on these markets where it is unlikely to ever make it back.

chart_aus_dogs170731

Us horses have produced again with 0.121% for July. June was 0.118%. Not vastly more markets traded than the Aus dogs but there are races that get as much money as the bigger UK dogs races and the higher liquidity gives more opportunities to trade. I guess these races are the televised ones?

chart_ushorse170731

I’ve been invited along to the Ebor festival at York races on the Saturday. I’m taking a bit more out of the bank for spend. It’ll be the first big race I’ve been to having only been to the races three times before – regular events at Doncaster, Ripon and Pontefract – so am looking forward to it.

Streaming and NSW

Mike left a comment on the previous post –

Always good to follow your progress. I’ve been trialing the new Gruss at home but have yet to let it loose with my Bots on my VPS. You say; “To note, the charts on the VPS are showing lower times than those at home” I’m reading that to mean “faster” times not “slower” times. What’s the bandwidth usage like with multiple sheets running?

As for the NSW markets, if you lay first it is theoretically possible to close out/green/red up your trade by laying out the other runners across the field, thereby not placing any back bets and avoiding the turnover charge. Albeit more complicated mathematics and process and it may not suit your trading style, but it is an option.

Thanks for the comment.

You are right, I meant faster times. Below is an image showing the charts whilst streaming from the VPS and from home, both monitoring same event at same time. Although the scales are different, I’ve added 100, 200 and 300 increments to highlight the difference. The VPS is generally lower times, faster, but more importantly, the consistency is better. For example, the VPS chart has 1 refresh greater than 300ms for this period, whereas at home I see 12 occasions of greater than 300ms.

gruss_stream11

In this next image I’ve tried to match the timings. Interestingly it looks like the home refresh is sometimes faster but this is just a consequence of refreshes coming in close after the refreshes that are notably slower (otherwise, if every refresh was slower, the market would drift away behind real time as the delay is compounded – mind boggling).

gruss_stream12

I’m not too sure how best to measure bandwidth in real-time but this is from the resource monitor in Windows Task Manager. This was taken with two sheets open. The first was a UK evening greyhounds with about 40s to off. The other was a US horse race from about 2 mins out (approx £5K matched). 2600 Bit per second is equal to 325 Byte per second (according to Google). Hope this helps, or let me know if there’s a better measure.

vps_bandwidth

Your possible solution to the NSW issue is very much outside the box and an excellent reminder of what we’re doing in the market. If you back one selection, you’re not only backing it, you’re effectively laying every other selection. Using this logic, as you say, it’s possible in theory to trade through the market without having to place a back bet. It does require a market with tight spreads. And also a large enough stake to be able to lay all other selections with as close to calculated values as possible. But it could work. Not likely for me though, off the top of my head, my initial stakes would be too low. I like the thinking though.

June ’17

The big event of June was the release of a beta streaming version of Gruss. As it is in development, the Gruss guys requested feedback and there were a number of bugs highlighted by users. We are now on the sixth release, which is good to know the program is being worked on and improved. The bug that I found related to moving on from suspended markets but after I fed back via the forum, a fix was quickly released. It took a little time for me to grasp the full effects of streaming. At first I thought the refresh was poor as updates were very random. The stream only updates when there is something to update, ie market activity, and doesn’t waste bandwidth by refreshing the same data repeatedly, as before streaming. This gives a refresh chart that can have quite large gaps between updates, especially on markets with some time to go before off. Now, with only a few minutes to off, the refreshes come in more than every 200ms. As there are no requests for price data, there’s no added delay. To note, the charts on the VPS are showing lower times than those at home. The request delay will still be relevant when placing orders in the market but no data is available on what it is.

Another major event, for me, was the changes to the Aus turnover eligibility. I posted about this here. I’m on with the coding around this. I have a section of code that only runs once when a market is selected and then isn’t run every refresh. I’m adding the NSW code there, which was straight forward for checking against the list of courses but tracking the traded back bets over a week is a little more complicated.

The UK dogs have done good this month. I’m considering adjusting the stake range to allow higher bets. I want to trial it on specific markets first. I’m thinking of those that are televised and tend to have much higher activity.

chart_ukdogs170630

An improved chart from the Aus horses compared to recent months. It’s nice to see a good return as I was beginning to lose patience with it and was considering stopping this bot. It’s been a long time since I ran Oscar on the UK horses as there was no value in it for me and I was thinking the Aus horses were going the same way. They still might, to be fair. But for now, with this, it will continue.

chart_aus_horse170630

US horses – better result than not trading them at all. There are some really well funded races that I’m missing, purely down to start times been well off. I’ll continue moaning about this point until I finally get a solution in place (I’ve had some good suggestions from you but the code don’t write itself, I should get on with it).

chart_ushorse170630

Definitely lower activity on the dish-lickers. An unfortunate loss keeping the return in the negative. Not much harm in continuing for now, I consider this my experimental contribution (why not?)

chart_aus_dogs170630

Aus Turnover Charge, Again

Back in Feb 2016 I wrote about Aus turnover charge. Since then the rules changed, meaning qualification for the charge was more difficult. However, I got an email last week from the friendly Betfair staff, warning me in advance of another rule change that happened today – 1st July (see Betfair Law). In brief, the charges apply to NSW races and now only 25 markets per week before charge eligibility. But which venues are in NSW? To avoid wearing out the map book, I tweeted @BetfairCS, who initially pointed me to the rules and regs. I persisted and a very helpful chap (or chapette) emailed over a list of venues in an Excel spreadsheet – result. For now I’ve set the bot to jump past the NSW markets, in future I’d like to code to allow trading up to the limits allowed. Here’s a link to the file. I also list below –

NSW Metropolitan  Abbreviation State Class
Canterbury Cant (AUS) NSW Metro
Randwick Rand (AUS) NSW Metro
Rosehill Rose (AUS) NSW Metro
Warwick Farm WFrm (AUS) NSW Metro
NSW Country Tracks Abbreviation State Class
Albury Alby (AUS) NSW Country
Armidale Armi (AUS) NSW Country
Ballina Bali (AUS) NSW Country
Bathurst Bath (AUS) NSW Country
Bowraville Bowr (AUS) NSW Country
Casino Casi (AUS) NSW Country
Cessnock Cess (AUS) NSW Country
Coffs Harbour Coff (AUS) NSW Country
Coonabarabran Cona (AUS) NSW Country
Coonamble Cnbl (AUS) NSW Country
Corowa Coro (AUS) NSW Country
Cootamundra Coot (AUS) NSW Country
Cowra Cowr (AUS) NSW Country
Dubbo Dubb (AUS) NSW Country
Forbes Forb (AUS) NSW Country
Gilgandra Gilg (AUS) NSW Country
Glen Innes GInn (AUS) NSW Country
Gosford Gosf (AUS) NSW Provincial
Goulburn Goul (AUS) NSW Country
Grafton Graf (AUS) NSW Country
Griffith Grif (AUS) NSW Country
Gundagai Gund (AUS) NSW Country
Gunnedah Gunn (AUS) NSW Country
Hawkesbury Hawk (AUS) NSW Provincial
Inverell Inve (AUS) NSW Country
Junee Jnee (AUS) NSW Country
Kembla Grange KemG (AUS) NSW Provincial
Kempsey Kemp (AUS) NSW Country
Leeton Leet (AUS) NSW Country
Lismore Lism (AUS) NSW Country
Moree Mree (AUS) NSW Country
Moruya Moru (AUS) NSW Country
Mudgee Mudg (AUS) NSW Country
Murwillumbah Murw (AUS) NSW Country
Muswellbrook Musw (AUS) NSW Country
Narrabri Nbri (AUS) NSW Country
Narrandera Ndra (AUS) NSW Country
Narromine Narr (AUS) NSW Country
Newcastle Newc (AUS) NSW Provincial
Nowra Nowr (AUS) NSW Country
Orange Orng (AUS) NSW Country
Parkes Park (AUS) NSW Country
Port Macquarie PMaq (AUS) NSW Country
Queanbeyan Quea (AUS) NSW Country
Quirindi Quir (AUS) NSW Country
Sapphire Coast Sapp (AUS) NSW Country
Scone Scne (AUS) NSW Country
Tamworth Tamw (AUS) NSW Country
Taree Tare (AUS) NSW Country
Temora Temo (AUS) NSW Country
Tuncurry Tunc (AUS) NSW Country
Wagga Wagg (AUS) NSW Country
Walcha Walc (AUS) NSW Country
Warren Warn (AUS) NSW Country
Wellington Well (AUS) NSW Country
Wyong Wyng (AUS) NSW Provincial