When scrape.py is run , It throws errors like Error opening url!!

[keerthankumar]

Is it still possible to scrape data from moneycontrol through this script in April 2018 or something changed in a website that script doesn't valid anymore?

[keerthankumar]

Hello Any response ?

[vintageplayer]

The script is executing correctly. Could you be more specific about the problem? "Error Opening URL!!" (Line 40 in the script) is mainly thrown due to timeout/connection issues. Also, it has been tested on python3 only, older versions of requests library might be an issue.

[keerthankumar]

Thanks for responding..

I was debugging from my end.. I could zero in that error is because of the signal module in windows platform. Which does not have attributes such as SIGALRM etc.?

Please tell me when u ran this did u ran on Unix platform or windows?

[keerthankumar]

one more question,
why do we require time_limit function ?

[vintageplayer]

It was tested on unix platform. Time function is there to handle request timeouts automatically. You can comment out the lines if required. The function could be simplified to:

def get_response(aurl):
	hdr				= {'User-Agent':'Mozilla/5.0'}

	try: 
		content 				= requests.get(aurl,headers=hdr).content
	except Exception:
		print("Error opening url!!")

	return content

[keerthankumar]

Will try this in windows and let you know.

Btw are you aware the legality of this code ? I mean is scraping data from Moneycontrol legal ?

[vintageplayer]

The robot.txt file disallows crawling certain endpoints, none of which this script accesses.
The terms of usage disallows sharing/broadcasting of site data (not specifically mentioning crawling/scraping). Also, it emphasises on stock call information.
This script is getting only the Profit&Loss and Balance Sheet information, which supposed to be public information for any publicly traded company.
To my best knowledge, there shouldn't be any issue with legality. Please let me know incase anything contradictory is found.

[jhawakshay]

Hi,

Does it work in Google chrome also? As, I am running the script, I am facing an error 'Access is denied'

[vintageplayer]

Hi,

Does it work in Google chrome also? As, I am running the script, I am facing an error 'Access is denied'

This script is browser independent. There is no browser automation required to run it. So Google chrome is irrelevant to running the script. This will run in the terminal python scrape.py or use an IDE.
Is that returning an 'Access is denied'?

[chinkoopataka]

Hi KeerthanKumar

Were you able to run it on Windows? I am getting the same "Error Opening URL" on spyder IDE on win 10. Please assist

[vintageplayer]

Hi, This was created a while back, maybe the endpoints have changed. Let me know the features required, maybe I can still patch it free time.

[chinkoopataka]

If you could please patch it, it would be awesome

[chinkoopataka]

For each company, it is saying "Data on ABC company doesnt exist anymore" where ABC is every company.
And what I don't understand is why is the program starting from the companies starting from letter 'H'

[vintageplayer]

For each company, it is saying "Data on ABC company doesnt exist anymore" where ABC is every company.
And what I don't understand is why is the program starting from the companies starting from letter 'H'

Money-Control/src/scrape.py

Line 268 in fca06a3

for link in links[8:]:

Is starting from index 8. Thus it's always starting with H.

[chinkoopataka]

Thanks, Didn't see it before.
Do you think you can look into this code dude?