In this blog post, I describe how I download a lot of images from Google images. I followed pyimagesearch's blog post so please give credits to his blog. His method has two steps:
- Step 1: The first step is to gather URL links of the images that appear in Google Images when you enter a query. pyimagesearch's blog post did this using Java Script. After running his ~10 lines of Java Script code, you will download a text file named urls.txt, that contains the URL link of the images.
- Step 2: The second step is to download images from each URL using Python.
I followed Step 1 and downloaded the urls.py. However, I wrote my own script for the Step 2. This was because his python scripts requires quite a few python libraries, and they are:
- from imutils import paths
- import argparse
- import cv2
- import requests, os
I could not pip install imutils in my environment so here I write a scripts that only require requests and os.
First, let's take a look at the urls.txt file. I renamed this file to be "urls - Hunter x Hunter anime.txt". (This is because "Hunter x Hunter anime" was my query for Google Image. Hunter x Hunter is my favorite anime show.) If you are interested in my .txt files, I pushed it in my github.
path_text = "urls - Hunter x Hunter anime.txt"
First let's look at the first 10 lines of the urls.txt and the number of lines.
o = open(path_text,"r")
url0 = o.read()
o.close()
## list, containing downloaded files
urls = url0.split()
print("The number of urls: {}".format(len(urls)))
print("____________________________")
for url in urls[:10]:
print(url)
Next download the images from each of the URL using requests.get(). I included the try catch as some requests fail with error messages. Runinng the following script create a folder data if it does not exist in the current directory and save images in jpg format.
import requests,os
loc_data = "./data/"
try:
os.makedirs(loc_data)
except:
pass
iimage = 0
for url in urls:
try:
f = open(loc_data + 'image{:05.0f}.jpg'.format(iimage),'wb')
f.write(requests.get(url).content)
f.close()
iimage += 1
except Exception as e:
print("\n{} {}".format(e,url))
pass
So you are done with downloading the images. You do not need the codes hearafter if your solo purpose is to downloading the data, but if you want to take a look at some of the images, here is the codes for peaking the first 9 images.
from keras.preprocessing.image import load_img
import matplotlib.pyplot as plt
fnames = os.listdir(loc_data)
fig = plt.figure(figsize=(10,10))
count = 1
for fnm in fnames[:9]:
img = load_img(loc_data +fnm,target_size=(400,400))
ax = fig.add_subplot(3,3,count)
count += 1
ax.imshow(img)
ax.axis("off")
plt.show()
