可扩展的Cookies池,用无头浏览器登录并生成cookie供给爬虫使用
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

49 lines
1.6 KiB

8 years ago
import json
import requests
from requests.exceptions import ConnectionError
from cookiespool.db import *
class ValidTester(object):
7 years ago
def __init__(self, website='default'):
self.website = website
self.cookies_db = RedisClient('cookies', self.website)
7 years ago
self.accounts_db = RedisClient('accounts', self.website)
7 years ago
7 years ago
def test(self, username, cookies):
8 years ago
raise NotImplementedError
7 years ago
8 years ago
def run(self):
7 years ago
cookies_groups = self.cookies_db.all()
for username, cookies in cookies_groups.items():
self.test(username, cookies)
8 years ago
class WeiboValidTester(ValidTester):
7 years ago
def __init__(self, website='weibo'):
ValidTester.__init__(self, website)
7 years ago
7 years ago
def test(self, username, cookies):
print('正在测试Cookies', '用户名', username)
8 years ago
try:
cookies = json.loads(cookies)
except TypeError:
7 years ago
print('Cookies不合法', username)
self.cookies_db.delete(username)
print('删除Cookies', username)
return
8 years ago
try:
7 years ago
test_url = TEST_URL_MAP[self.website]
8 years ago
response = requests.get(test_url, cookies=cookies, timeout=5, allow_redirects=False)
if response.status_code == 200:
7 years ago
print('Cookies有效', username)
8 years ago
else:
print(response.status_code, response.headers)
7 years ago
print('Cookies失效', username)
self.cookies_db.delete(username)
print('删除Cookies', username)
8 years ago
except ConnectionError as e:
7 years ago
print('发生异常', e.args)
7 years ago
if __name__ == '__main__':
WeiboValidTester().run()