Probably you also should make plans not to live in California - this state is also experiencing quite severe droughts year after year - probably will be same fate as Arizona in maybe couple decades?
This is unbelievably awesome! I wish I had such material and teacher like Bartosz Ciechanowski when I was learning IC engine during my undergrad for mechanical engineering. Thanks a million to Bartosz wherever you are and whoever you are! Simply wonderful! Thank you, thank you, and thank you!
I frequently communicate architecture with diagram that simply contains following three things:
- boxes (rectangles with text in them)
- arrows connecting these boxes
- and some brief text
to convey any architecture/ideas
Apart from this, sometime I also use "sequence diagrams" - vertical lines representing systems/modules to illustrate the sequence of flow of any system/user action.
How does Betfair get this data into their system? Who provides them with this data or they too write their own web scraping application to collect and consolidate the data?
Probably it is manual because they need to take bets and decide what to pay out on. If they get it wrong, they lost a lot of money and/or credibility. To be totally sure, DYOR.
There is no way they can trust scraped data. Especially when you can have different matches with slightly different names. For example U21 version of A vs B for a givens sport (learned that the hard way when arbing!).
Thanks. But i am still curious to know how does ESPN gets all the schedule for all sports in their persistence store? Does NFL, NBA, IPL or other sports league provide them with APIs to fetch data or does ESPN too has web scraping scripts that goes after NBA/NFL/MLB websites to get the data they need or do they have people actually doing manual data entry for all of this information via some UI into their own database?
I know a guy who works in sports betting software, a pretty large player. They have direct access feeds that they pay a lot for bulk access, to cbs sports, nbc, etc. I dont know how much but its no small sum. ESPN does not use web scraping for live stats- they use high speed apis that get info right from the field (that either their crew gathers or cbs gathers and they pay for access). The overwhelming majority is not manually entered.
I also do some scraping in my spare time and i can tell you that large companies will have... precautions in place to prevent you from scraping their by-the-minute sports stats they pay millions for. Id also say right now scraping is a gray market, airlines in particular have proven that they are willing to fight for their (public) data, so tread carefully.
Scraping is a problematic way to do this too, because how can you verify what is up to date? You have to rely on one single authoritative source, otherwise youll be relying on some stat calc which “guesses” what the right stat is. Additionally, if you take live data that multiple large businesses feel they own, now they can split the lawyering fees on you and even if they lose, it will be cheaper for them together.
The historical content probably would be trivial to get, and its hosts most likely dont care if you have it. its the recent/live data which may be particularly difficult to take. In my professional opinion this is two separate projects- a paid up to date version with live stats and a free historical wiki.
Thanks. I do not have any intention to do web scraping. Let's reduce the scope of my request: I want to do two things - 1) Get the schedules of the upcoming (future) matches in most of the popular sports across globe - football, soccer, cricket, tennis etc. How does ESPN or other sources get this? Do they get this information 3 months/6 months in advance from all these sports organizations? Or do they simply have people to enter this data manually in their own system? 2) I want to keep history of who won/lost the particular match and any other interesting tidbits about the match that already took place. So I am not planning on doing any web scraping and neither I plan to offer any live feed API or data. So my question is - how do I accomplish my above stated 2 goals efficiently without doing web scraping? Thanks.
I come myself from middle class Brahmin family in Maharashtra, have a engineering degree from a mediocre school in Maharashtra. If at all I witnessed something negative is snobbishness from people who graduated from better school than me. I have been in silicon valley for over 25 years now. I worked in 2 startups and 4 mega corporations/public companies with friends/superiors/colleagues (some of whom are Brahmin IIT graduates) from various parts of India - I personally never asked anyone's last name to find their caste and neither I witnessed anyone making any derogatory remarks about anyone's caste/religion. I have never experienced or seen any of this that is being reported in this article. It is truly shameful and disgusting if it is happening now or before. Let's stop this right now and right away!
I appreciate people like you. But the thing is, if you are an upper caste person then it is very less likely that you'll be discriminated against as far as caste is concerned.
Overtly discrimination will happen less but covertly ot happens a lot.
But the good news is that not everyone is like that so we are slowly progressing
