Xtykutl

Working on a little web scraping program to get some data and help me make some bets.

Ultimately, I want to parse the "Trends" section under each game of the current week on pages like this (https://www.oddsshark.com/nfl/arizona-kansas-city-odds-november-11-2018-971332)

My current algorithm:

1. GET https://www.oddsshark.com/nfl/scores
   


2. Parse the webpage for the little "vs" button which holds links to all the games


3. Parse for the Trends

Here's how I started:

from bs4 import BeautifulSoup

import requests



url = "https://www.oddsshark.com/nfl/scores"

result = requests.get("https://www.oddsshark.com/nfl/scores")

print ("Status: ", result.status_code)



content = result.content

soup = BeautifulSoup(content, 'html.parser')



print (soup)

When I look at the output, I don't really see any of those links. Is it cause a lot of the site of javascript?

Any pointers on the code/algorithm appreciated!

You can use the internal API this sites uses to get all the links & iterate over these to get the trends info which is embedded in a script tag with id:gc-data :

import requests

import json

from bs4 import BeautifulSoup



r = requests.get(

    'https://io.oddsshark.com/ticker/nfl', 

    headers = {

        'referer': 'https://www.oddsshark.com/nfl/scores'

    }

)



links = [

    (

        t["event_date"], 

        t["away_name"], 

        t["home_name"], 

        "https://www.oddsshark.com{}".format(t["matchup_link"])

    )

    for t in r.json()['matchups']

    if t["type"] == "matchup"

]



for t in links:

    print("{} - {} vs {} => {}".format(t[0],t[1],t[2],t[3]))

    r = requests.get(t[3])

    soup = BeautifulSoup(r.content, "lxml")

    trends = [

        json.loads(v.text)

        for v in soup.findAll('script', {"type":"application/json", "id":"gc-data"})

    ]

    print(trends[0]["oddsshark_gamecenter"]["trends"])

    print("#########################################")

The reason you don't see those links is that they're not in the response that requests receives. This is very likely for one of two reasons:

1. The server recognizes that you are trying to scrape the site with a script, and sends you different content. Usually this is because of the User-Agent set by requests.


2. The content is added dynamically via JavaScript that runs in the browser.

You could probably render this content using a headless browser in your python script and end up with the same content you see when you visit the site with Chrome et. Per (1) it might be necessary to experiment with the User-Agent header in your request also.

The data is loaded via javascript to the trends table, but is actually included in a script tag inside the html that you receive. You can parse it like this:

import requests

import json

from bs4 import BeautifulSoup



headers = {

    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0'

}



response = requests.get('https://www.oddsshark.com/nfl/arizona-kansas-city-odds-november-11-2018-971332', headers=headers)



soup = BeautifulSoup(response.text, "lxml")



data = json.loads(soup.find("script", {'id': 'gc-data'}).text)

print(data['oddsshark_gamecenter']['trends'])

Outputs:

{'local': {'title': 'Trends'}, 'away': [{'value': 'Arizona is 4-1-1
ATS in its last 6 games '}, {'value': 'Arizona is 2-6 SU in its last 8
games '}, {'value': "The total has gone UNDER in 8 of Arizona's last
12 games "}, {'value': 'Arizona is 3-7-1 ATS in its last 11 games on
the road'}, {'value': 'Arizona is 2-4 SU in its last 6 games on the
road'}...