Automatically Checking in Bulk Whether Links Open

#Background

Scenario: you've taken over a website with a long history, but its SEO ranking is pretty weak. That may be because there are too many broken links, which leads search engines to judge that the site's sitemap is unreliable, and ultimately affects the site's ranking.

Once you've pulled the sitemap file, you can't possibly click through every link by hand, can you? That's when you need an automated script that can batch-check whether the links on the site open.

#Prerequisites

Get the website's sitemap file, which lists all of the site's links. Alternatively, you can import a CSV file manually. The format looks roughly like this (it's really just a list of full URLs):

https://estruturarte.com.br/pickup/F1850555
https://ns.hashva.com/mah.php?wqyaesdppr/sj-371792.html
https://classbclass.ir/content.php?pickup/K1178460
https://forexreprogramming.com/pickup/I1752252
https://eglow.mx/pickup/K1221646

#Code

The code needs to run in a Python environment.

import csv
import requests
from tqdm import tqdm

urls_file = 'output_10.csv' # 填写你的文件路径
result_file = 'result/output-10-result.csv' # 填写检测结果的存放路径

with open(urls_file, 'r') as f:
    reader = csv.reader(f)
    urls = [row[0] for row in reader]

results = []

for url in tqdm(urls):
    try:
        response = requests.get(url, timeout=5)
        if response.status_code == 200:
            result = 'success'
        else:
            result = 'fail'
    except requests.Timeout:
        result = 'timeout'
    except:
        result = 'error'

    results.append([url, result])

with open(result_file, 'w') as f:
    writer = csv.writer(f)
    writer.writerow(['URL', 'Result'])
    writer.writerows(results)