Tag Archives: Comment

Outsourcing theft

Seamus asked –

How does using a VPS affect your risk of hackers stealing your strategies?

The main risk from using a VPS is not from hackers but from an insider ie an  employee of the VPS provider. Some random could target the VPS provider but I’d guess there’s more valuable data than my penny bot.

There are steps to take to improve security. Google “setting up a VPS” for some tips. Also password protect excel code. This is weak but combined with an event handler that clears certain settings on close and open events would render the bot useless. As with all things, if someone really wants it they’ll get it, whether that’s hacking your home pc, your VPS or smashing your back door in and legging it with your computer.

If you’ve got a very profitable bot then I don’t see any need to move to a VPS, you’re already winning. If, like me, your main point when people ask “how’s the betting going?” is to say “well, I don’t lose”, then move to a vps, save money on the electric bill and have less connection-loss, enjoy the feeling of playing in the big league and don’t worry about someone twoc-ing your ok bot.

Advertisements

… and this

For clarity on the discussion, I added the following image to the Gruss forum thread.

It includes the closing bets (a mirror as expected) and the p&l for the period. The moving average is just to give a clearer view of the trend. (And colour is different to first chart.)

aus_horse_bk_ly_pl6_mth

Values.

Negapo and Captain Sensible commented/discussed over on the Gruss forum (click here to have a look) referring back to my discussion with Steve that started here. It’s about value and where we find value. Both discussions are interesting. Here’s how I see value and what it means to my bot.

True value

The true value (true odds/true price/etc) of any horse/dog/yacht can never be known before the event. The fact is a horse will either win or lose (except in a dead heat but let’s ignore that for now). The closest thing we have to true value is the starting price (SP). If you were to back (on Betfair) every favourite horse at it’s SP, I’d expect you would be on average around break-even, having winning/losing runs, but nothing consistent (your wallet would decrease due to commission paid). If this was not the case and SPs were always overpriced for example, then this would be known and people would exploit them, which in turn would effect change on the SPs, pressuring them towards true value, therefore making the SP near to true value. It’s a system constantly being balanced by it’s participants (the crowd).

Value bet

You’ll never know the true value of a single runner. You have your views though and decide that dog number two in the 15:35 has a true value of 4.0, the market is currently at 5.0, you back it with a stake of £10. It wins. Were you right or wrong to price it at 4.0?  Again, you’ll never know, not on that one outcome. After the win, and after the fact, the true value was 1.0, because it won. In the unlikely event that you always pick winners, not one loser, then stop reading this, you are unique and very rich (or you’re trying to sell something). You’d also be able to say the true value of any of your selections is 1.0, because they always win. In all other scenarios though, you’ll know if you’re finding value bets because you’ll be making profit in the long term. If you repeatedly back at 5.0 when the true value is 4.0, you’ll see it in your results. You wont win every bet, but you’ll be on the right side making profit.

Value trade

Finding value when trading is, or can be, different. Making profit will mean at least one bet in a trade is at value but this just has to be, as you’ll be making that trade across, or on one side of, true value, with both a back and lay position. I find it easier to explain this using some excellent graphics indicating back and lay points.

The trade in the image below, with respect to outcome, the lay bet is at value, the back bet isn’t. The result is profit.

layatval

The trade in the image below, with respect to outcome, the back bet is at value, the lay bet isn’t. The result is profit.

back_at_val

The trade in the image below, with respect to outcome, both the back and lay bet are at value. The result is profit.

The trade in the image below, with respect to outcome, the lay bet is at value, the back bet isn’t. The result is loss.

lay_at_val_lose

The trade in the image below, with respect to outcome, the back bet is at value, the lay bet isn’t. The result is loss.

back_at_val_lose

The trade in the image below, with respect to outcome, both the back and lay bet are not at value. The result is loss.

bk_and_ly_no_val

From these professional quality graphics we can see two things for sure –

  1. If both back and lay bets are at value then you have a profit.
  2. If both back and lay bets are not at value then you have a loss.

In the other masterfully created images where one bet is at value and the other isn’t, it is the relative price of the back and lay that determines profit or loss.

What’s it mean to me?

This brings us back to the point I was challenging last year regarding API crashes – “Every bet you place with a bot should be sent because you believe it to be value at that time and it should be allowed to stand on it’s own merits.” – Steve.

Looking again at the first image –

layatval

If my entry point is the lay at 2.34 then not closing would leave me with a value bet. If my entry point is the back at 2.42 then not closing would leave me exposed with a less than value bet. However, I see statistical value in backing at 2.42 and I’m predicting a move to 2.34 due to market conditions, with no regard to the runner/selection/sport. I need to close the trade to profit. (These figures are for illustration, in reality I’d be expecting a move from 2.42 to 2.40, one tick only.)

I have no idea at all if my bot is entering the market at value with regards to outcome. My only concern is the short-term price movements. My bot is scalping, chasing pennies, harvesting small profit from others intended trades, jumping in the middle, whatever you want to call it. I have no care for the outcome or the sport, only the market and it’s behaviour. Therefore my requirement is to close out every time, for profit or loss, within seconds of entry. I am aware this type of trading is disliked by some, such is life.

Out of interest and in response to a comment by Captain Sensible, below is a chart of the results (P&L) of all the back entry bets placed over 6 months on the Australian horses. As can be seen I’d currently be over £300 up, hitting a high of over £400 and a low of around -£300. But there is no consistency to be relied upon here. This chart suggests to me that I’m not relying on my entry being at good value with regards to outcome. Closing all these trades resulted in a steady consistent profit.

aus_horse_bk6_mth

 

Updated updating and simply greening

First, Liam replied to July ’17

You can set the conflation during the market subscribe request, defaults to 0 but you can choose how often you want to receive it. It’s odd because it’s a handy feature but none of the trading apps seem to use it.

I know betfair use a piece of open source software called Kafka for managing streaming updates but there is probably something else handling slow consumers (conflation). Sockets are complex but at a high level it is my understanding that once the receivers buffer is full the server is told this which then allows the server to halt sending more data. I believe it is then a case of updating that ‘halted’ data to prevent the consumer receiving ‘stale’ data.

Hopefully that makes more sense.

 

Thanks Liam, it does. I appreciate this is more in-depth stuff and beyond most of us but I am interested in these things. It’s the sort of detail that gets exploited by those in the know before anyone else cottons on. I wonder if the lack of utilisation by trading apps is because setting it for optimal performance would be very specific to an individual socket, the machine’s processor, what else it was doing at the time and the level of market activity?

test_passed_stamp

Now, bench testing complete, I’ve put the new Oscar live on the Aus horses only (staged implementation, sounds good). As mentioned in the last post, there are two areas where significant changes have been made to the code.

I trialled two different methods of navigating the markets. One is quite basic and simple code, the other more complex but I thought it would handle any troublesome delays better. It was over complicated though and as I worked through the errors whilst testing, it became bloated and confusing. Impact wise, the simpler code is only one extra step on an if-then-else statement. And it’s the same amount of steps as the old code. I’ve set it to update after a set time, 1:45am, then select first race in list shortly after. I use the day number, as integer, to check if the process has completed today. Previously the list update was performed within a set time period, which was fine when there were regular updates at the desired refresh rate. But with streaming, if there’s a market loaded that’s a few day’s off, as is sometimes the case for big events, there are no regular refreshes. This can mean that the set time window see’s no updates to run the list refresh code.

That change is almost not needed because of the next change. I’ve coded to turn full streaming off when not within the trading time zone. This forces regular refreshes. The above change will still come into use though if there are any issues causing a delay (can’t think what, but the code is better now, I’m sure).

Finally for this update, I’ve employed the green up trigger to replace my code. The module is less than a quarter of it’s previous size and the trigger takes care of cancelling any unmatched bets, submitting a greening trade and then chasing every second until matched. Splendid.

July ’17

Liam commented on Streaming and NSW

Its important to note that latency from streaming is a function of how quickly you can process the updates. If your CPU is unable to process the updates quick enough betfair will apply conflation which is what you are probably seeing. Of course network latency can impact this but its is more likely to be due to CPU / algorithm you are using to process the stream.

“””
Writes to your connection are directly effected by how quickly you consume data & clear your socket’s buffer
Consuming data slowly is effectively identical to setting conflation.
If you receive conf=true flag on a market – then you are consuming data slower than the rate of deliver
“””

Thanks for the comment.

This is something new to me and trying to find further info doesn’t reveal much. I haven’t seen any reference to it in the Gruss error log. Although I’m not entirely sure I understand this correctly. The statement in the quotes talks of “effectively setting conflation” by consuming data slowly. But who sets conflation? Which I guess to mean that updates are delivered in groups, merged, rather than individually. Is it something you can do from this side? Or is it a decision made by Betfair? But how would they know how you’re processing the data? My understanding of streaming is that it arrives at your computer as is, stored in a local cache(?), then processed by the program/bot. Would it be delivered/streamed to the computer conflated? Or conflated in the local cache for processing? I don’t know but it sounds interesting and any more information would be appreciated.

I’m only using Gruss at the moment as I’m not really excited by the prospect of coding for streaming in VB. My ventures into VB continue but not with Betfair. I finished a basic lay stake calculator and am now on with a more advanced version, trying to get a rolling list to display neatly. Also tested some scraping code and played with excel controls. I find Visual Studio so much more advanced than the VBA IDE that coding/debugging is a lot clearer and faster.

Back to trading. I’ve two test bots in coding at the moment. One is a reworking of a bot I’ve trialled on and off. Simply, it scrapes data and then trades. The thing is, each time I go back to it I have to change a chunk of code as the websites, as is always the case, alter their layout or swap some data from html to a type that doesn’t scrape easily (javasript/flash?). On the bright side it does mean that I learn some new code or technique.

The second test bot is an update to Oscar to take in the recent changes introduced with streaming. There is two parts to this – market navigation and greening. I’m having problems with the bot stopping on suspended/closed/abandoned events and have missed a number of days across the different bots. Although some change was made by the Gruss team, improving the situation, it is still happening. So to help from my end, I’m changing the way my bots refresh the markets (list of available events for each bot) which are held in the Quick Pick List. This will also include a change to how the bots jump from one market to the next; switching off full stream, which forces a specific refresh rate, prior to leaving an event, then switching full stream on once a new market is loaded and certain conditions are met.

EDIT: Team Gruss have said they’re on with fixing this problem 🙂

A new feature now available is the eagerly awaited and much anticipated greening trigger (it really has been wanted for ages and requested loads on the forum). My greening code is quite messy with multiple attempts to green. It’s been a crude hack from the early day’s that I’ve bodged to work rather than start fresh and I’m happy to see it go to be replaced by a simpler solution.

Oscar now and the UK dogs. Hmm. It is positive, that’s about it.

chart_ukdogs170731

Aus horses. Also positive…

chart_aus_horse170731

Aus dogs. The story of the 6p I wish I still had. Seriously though, there just isn’t the liquidity so the result could be anywhere to be fair. I stopped this bot at the beginning of August with a profit of 12p since May. I don’t see any reason to risk a big loss on these markets where it is unlikely to ever make it back.

chart_aus_dogs170731

Us horses have produced again with 0.121% for July. June was 0.118%. Not vastly more markets traded than the Aus dogs but there are races that get as much money as the bigger UK dogs races and the higher liquidity gives more opportunities to trade. I guess these races are the televised ones?

chart_ushorse170731

I’ve been invited along to the Ebor festival at York races on the Saturday. I’m taking a bit more out of the bank for spend. It’ll be the first big race I’ve been to having only been to the races three times before – regular events at Doncaster, Ripon and Pontefract – so am looking forward to it.

Streaming and NSW

Mike left a comment on the previous post –

Always good to follow your progress. I’ve been trialing the new Gruss at home but have yet to let it loose with my Bots on my VPS. You say; “To note, the charts on the VPS are showing lower times than those at home” I’m reading that to mean “faster” times not “slower” times. What’s the bandwidth usage like with multiple sheets running?

As for the NSW markets, if you lay first it is theoretically possible to close out/green/red up your trade by laying out the other runners across the field, thereby not placing any back bets and avoiding the turnover charge. Albeit more complicated mathematics and process and it may not suit your trading style, but it is an option.

Thanks for the comment.

You are right, I meant faster times. Below is an image showing the charts whilst streaming from the VPS and from home, both monitoring same event at same time. Although the scales are different, I’ve added 100, 200 and 300 increments to highlight the difference. The VPS is generally lower times, faster, but more importantly, the consistency is better. For example, the VPS chart has 1 refresh greater than 300ms for this period, whereas at home I see 12 occasions of greater than 300ms.

gruss_stream11

In this next image I’ve tried to match the timings. Interestingly it looks like the home refresh is sometimes faster but this is just a consequence of refreshes coming in close after the refreshes that are notably slower (otherwise, if every refresh was slower, the market would drift away behind real time as the delay is compounded – mind boggling).

gruss_stream12

I’m not too sure how best to measure bandwidth in real-time but this is from the resource monitor in Windows Task Manager. This was taken with two sheets open. The first was a UK evening greyhounds with about 40s to off. The other was a US horse race from about 2 mins out (approx £5K matched). 2600 Bit per second is equal to 325 Byte per second (according to Google). Hope this helps, or let me know if there’s a better measure.

vps_bandwidth

Your possible solution to the NSW issue is very much outside the box and an excellent reminder of what we’re doing in the market. If you back one selection, you’re not only backing it, you’re effectively laying every other selection. Using this logic, as you say, it’s possible in theory to trade through the market without having to place a back bet. It does require a market with tight spreads. And also a large enough stake to be able to lay all other selections with as close to calculated values as possible. But it could work. Not likely for me though, off the top of my head, my initial stakes would be too low. I like the thinking though.

A comment, an icon, rounding and an offer.

US Horse Racing Off Times have been a problem for bot developers since the year dot. The US off times are just a guide and are not religiously adhered to like the rest of the world. Initially I got around this by polling the Time to Post stat that gives a guide of when the off time is due. This value can be scraped from a number of different sites. Even this was a little hit and miss. What I eventually landed on was waiting till the overround was less than 105% to indicate that the race was about to go off. Works quite well in sparse US markets where all the money comes in at the end before the off.

Thanks for this comment. I’ve looked into scraping times and have found a few sites. Not made any attempts to integrate yet. The idea of watching the overround is very good and something I hadn’t thought of. This goes on the to-do list, thanks.

 

 

I’ve been coding an app in Visual Studio to replicate a spreadsheet I’d made when I was doing the matched betting thing. I added an icon and thought I’d add it to this site, so if you haven’t noticed, it should be at the top of the address bar and look like this –

favicon3

I like the simple look, which is fortunate. And it should show up as different to WordPress in any bookmarks, or in the blog roll on green-all-over (It’s been quiet over there recently).

 

The app I’ve made calculates the lay stake at given odds. A problem I encountered whilst testing my little app was a difference between what I calculated and what I was seeing on the Betfair site. It’s only small but at large stakes it’s noticeable and any discrepancies are worrying, suggesting an error in the formula used. My error came down to rounding. Initially I was just rounding the result only to 2dp. Each step of the formula requires rounding to 2dp to keep it accurate. At no point can you place a lay stake of £50.083333. Attention to detail is key, my mistake.

 

Talking of attention to detail, I recieved an offer to try out a VPS in Dublin with less than 3ms delay. That’s extremely fast for sports trading and probably as good as it gets without co-location (something I’m sure goes on). I had looked at getting a Dublin based VPS when I was setting up but was not willing to pay the obvious premium rate. The offer I received came via Twitter and the first thing anyone does is look at the website of the people making the offer. I remember this company was looking for people to trial its VPS about a year ago. Their website still has some pages that haven’t been altered from the template settings – the blurb you get when you start a website from a pre-made layout offered by these site builder outfits. There’s a lot of trust when putting your bots onto a VPS and you want to know you’re in safe hands that pay attention to detail. I declined the offer.