Is it legal to scrape Twitter?

I've never done anything about this because I've always just presumed I couldn't.

For example I'm working on a piece that includes an interview with somebody. There's no real way I can include the name of my interviewee without it simply not appearing. Twitter would be an easy way to include what my question was, but unless I include it with their explicit consent on a Twitter profile page I'm not sure I can.

Is this legal? Is this even ethical? Should it even matter? It depends on whether they are a public figure or have a celebrity or similar level identity. But if they are a private individual/entity, no. If you want their permission, try asking them or setting up your own web site for the purposes.

As far as social accounts go, you can do a text search and find people who've mentioned you or what you do by running a string like. "@jeffclausen Twitter". This will work if they're using their real name. If not, try something similar.

As long as you include a link to the person's Twitter account, you should be fine under twitter's TOU. If their profile doesn't specifically give you access to it, you're not allowed to grab their personal information without asking them first. While scraping or snooping around Twitter may get you some extra information, it's usually not a wise idea, to say the least.

I think it is a really grey area. There is nothing technically stopping you from doing so, other than being potentially subject to legal actions if you make use of the info. (And even here, some jurisdictions, eg Spain, have no real-life criminal sanctions on these types of activities). There is also no explicit restriction in any law I know of preventing someone from doing so.

It is certainly unethical though, in my opinion. To me, scraping the data off someone else's website is basically a clear case of theft, plain and simple. Of course, there are plenty of other reasons why it may be unethical to do something, such as: lack of permission, violation of the robots.

What is the tool to scrape tweets?

I have tried
#!

/usr/bin/python. Import urllib2. Def downloadtweets(url): result = urlopen(url). content = result.read() return content. Tweets = downloadtweets(""). But it doesn't work well. Some times it works well but some times it has wrong results. How to scrape tweets correctly? Thank you.

The only way to make sure your scrapes are returning correct results is to use a tool that checks what you are scraping for. Many times a scraping script will find a tweet that does not return any results or might see the tweet a different way then a regular user.

Some suggestions: Try a site like www.tweetometer.com that can tell you what is and isn't good data. The best data on a twitter search is usually in between 15k - 50k or so.

Use a service that can get your data for you at a low rate like www.textnow.com
Do you need everything or just a limited subset? That might determine what kind of tool is most suitable for you.