Metadata-Version: 2.1
Name: crawlerHelper
Version: 0.0.1
Summary: tools which can easy the process of making crawler
Home-page: https://gitlab.com/snippets/1873717
Author: Wyatt Huang
Author-email: p@hty.email
License: UNKNOWN
Description: # Introduction:
        
        This class can be used in crawler project. And it contain two function:
            
            1. proxies IP
            
            2. hide header
        
        If also used pkl cache to speed up the abstract information from IP Pool
        cache will be expired in one day
        
        How to use:
        create new crawlerComponent object, and then,you can use:
        
            1. get_an_ip: get a random IP
            
            2. get_a_header: get a random header
            
            3. updateIpLib: update the ip library (original set as once per day)
            
        methods to use
        
        ---
        > Class Version: 2.0.0 in https://gitlab.com/snippets/1873717
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Description-Content-Type: text/markdown
