Skip to content
Gold Price, a Web Scraping Project
  • AI Chat
  • Code
  • Report

    • Gold Price, a Web Scraping Project

    In this workbook I will show some data scraping techniques to get real time quotes of precious metals from web in a fast, free and automated way! We will see how to extract Gold, Silver & Platinum price from Kitco.com and convert it from "Dollar / Troy Ounce" to "Euro / Gram" using Google Finance currency rates.

    This Python code can be used, for example, to automatically update the price of precious metals on your website or extract a .CSV with real time quotes.

    Disclaimer: As stated in Kitco.com Terms of Use, data seen and/or extracted with this code are intendended to be used for informational purpose only.

    1. Configuration & Functions Definition

    In this section we will import libraries and define the functions used to extract and convert data.

    Import libraries.

    import requests
from datetime import datetime

from bs4 import BeautifulSoup
import pandas as pd

    Define the function to get "Dollar / Euro" currency rate from Google Finance.

    def get_usd_eur():
    
    global usd_eur
    
    # Connect with Google Finance
    URL = 'https://www.google.com/finance/quote/EUR-USD'
    HTML_PAGE = requests.get(URL).text
    soup = BeautifulSoup(HTML_PAGE, 'html.parser')

    # Extract "Euro / Dollar" rate
    eur_usd = soup.find_all('div', attrs = {'class': 'YMlKec fxKbKc'})
    eur_usd = [float(x.string.replace(',', '')) for x in eur_usd][0]

    # Convert "Euro / Dollar" rate in "Dollar / Euro"
    usd_eur = round(1 / eur_usd, 5)
    
    return usd_eur

    Define the function to get bid & ask value of precious metal in "Dollar / Troy Ounce" from Kitco.com

    def get_metal_value(URL):
    
    global bid, ask
    
    # Connect with Kitco.com
    HTML_PAGE = requests.get(URL).text
    soup = BeautifulSoup(HTML_PAGE, 'html.parser')

    # Get precious metal value in "Dollar / Troy Ounce" and convert it from string to float
    bid_string = soup.find_all('span', attrs = {'id': 'sp-bid'})
    bid = [float(x.string.replace(',', '')) for x in bid_string][0]

    ask_string = soup.find_all('span', attrs = {'id': 'sp-ask'})
    ask = [float(x.string.replace(',', '')) for x in ask_string][0]
    
    return bid, ask

    Define the function to convert precious metal value from "Dollar / Troy Ounce" to "Euro / Gram".

    def convert_metal_value():
    
    global bid_final, ask_final
    
    # Convert precious metal value in "Euro / Gram"
    t_oz = 31.1034768
    bid_final = round(bid * usd_eur / t_oz, 5)
    ask_final = round(ask * usd_eur / t_oz, 5)

    return bid_final, ask_final

    2. Data Scraping & Export

    In this section we will store Gold, Silver & Platinum value in a DataFrame so it can be printed on console or exported in .CSV file format.

    Extract and convert data.

    URL_dict = {'Gold': 'https://www.kitco.com/charts/livegold.html',
            'Silver': 'https://www.kitco.com/charts/livesilver.html',
            'Platinum': 'https://www.kitco.com/charts/liveplatinum.html'}

header_ = ['metal', 'bid', 'ask', 'datetime']

list_ = []
datetime_ = datetime.now()

# Get and convert values from web
for key, value in URL_dict.items():
    get_usd_eur()
    get_metal_value(value)
    convert_metal_value()

    list_.append([key, bid_final, ask_final, datetime_])


# Print values
data = pd.DataFrame(list_, columns = header_)
display(data)


# Export data in .CSV
data.to_csv('metals_' + str(datetime_) + '.csv', sep = ';', decimal = ',', index = None)