Public/imdbscrapper
1
0
Fork 0
mirror of https://github.com/zebrajr/imdbscrapper.git synced 2025-12-06 00:20:21 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Added env variables for startURL, endURL, steUpCycle, nrProcesses to docker

Browse Source
This commit is contained in:
Carlos Sousa 2021-05-03 15:01:27 +02:00
parent 8d6a3aec8a
commit 301de775a8
2 changed files with 10 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docker-compose.yml
View File

@ -4,6 +4,11 @@ services:
container_name: imdbscrapper
image: zebrajr/imdbscrapper:latest
volumes:
- '${PWD}/src:/opt/imdbscrapper:ro'
#- '${PWD}/src:/opt/imdbscrapper:ro'
- '${PWD}/storage:/opt/storage'
user: 1000:1000
environment:
- START_URL=1000
- END_URL=0
- STEPUPCYCLE=100
- PROCESSES=5

8
src/scrapper.py
View File

@ -124,10 +124,10 @@ def imdbscrapper(startURL, endURL):
def main():
cls()
nrProcesses = 5 # Number of Processes to start in parallel
startURL = 10000 # Starting Number
endURL = 0 # Ending Number
stepUpCycle = 100 # How many numbers will be checked in each cycle
nrProcesses = int(os.getenv('PROCESSES', 5)) # Number of Processes to start in parallel
startURL = int(os.getenv('START_URL', 10000000)) # Starting Number
endURL = int(os.getenv('END_URL', 0)) # Ending Number
stepUpCycle = int(os.getenv('STEPUPCYCLE', 100)) # How many numbers will be checked in each cycle
stepUpProcess = int(stepUpCycle / nrProcesses) # Divides the numbers to be checked in each cycle by the total number of processes
# Eg: 5 Processes running each cycle. 100 Numbers each cycle. 20 Numbers per process