How do I extract information from Reddit?

How to use scraped Reddit data?

So I have been scraping Reddit and looking for interesting data. There is a subreddit called /r/talesofreddit where users share stories of what they have learned about the site or how they used a specific feature. For example, this one.

I was interested in this kind of stories so I thought to give it a try. First I searched for a list of stories that had learned in their title. I found something useful in the following query: And I ended up with about 12000 stories. They were all in the form of posts in a subreddit called /r/learned.

After that, I tried to find stories that had the phrase reddit secret in their title. This was a bit tricky as the title also could have the word secret. I wrote the following script in Python:

Import requests from bs4 import BeautifulSoup r = requests.get(') soup = BeautifulSoup(r.text, 'html.append(story.text)

This gave me about 30 stories. It looks like there are plenty of other ways to find interesting stories.

How to build a story list? Once I have a list of stories, I can look at each one and see if it has a description or some extra info. In that case I will scrape the text and then see if the story can be used for anything. The more info we have, the better.

So let's look at this example from the link above. A reddit user named SxnFwSrQyG has used Reddit to learn about a new feature: I think it was the subreddit feature. This is an example of a story that I found from the search results.

How do I extract information from Reddit?

I'm working on a little project in Python and I'm having trouble getting information from a subreddit. I want to extract something along the lines of: Title. Number of posts. Number of upvotes. Number of downvotes. List of posts sorted by upvotes. And then a list sorted by downvotes. I'm having trouble finding good documentation on how to go about this, but I'm aware that I'll be looking at urllib or mechanize for scraping content. This may seem like it's overkill, but I'm working with the /r/all feed myself right now. Here is a nice tool called ScraperWiki that will help you with a lot of the work you have to do.

For example, suppose the link you want to scrape is at Then with the following code, you can access all information you need: import re. Import sys. Import urllib2. Import cookielib. From BeautifulSoup import BeautifulSoup. From BeautifulSoup import Comment. Username = 'yourusername'. Password = 'yourpassword'. Def login(): auth = urllib2.HTTPPasswordMgr() auth.addpassword(None, 'reddit.com', 'yourusername', 'yourpassword')
# If you don't need to actually write cookies, uncomment the last line of this function. # auth.addtoken('yourusername', 'yourpassword', '/', '') handler = urllib2.HTTPBasicAuthHandler(auth) opener = urllib2.buildopener(handler) urllib2.installopener(opener) if not sys.urlopen(') r = response.read()