Let TweetNest Accept JSON
3200.
It's a sad reality that we've been living with for years now. I hit my 5 year anniversary with Twitter two days ago, yet I can only go back two weeks if I wanted to search for anything, which is obviously a limitation of the current API.
This scared me so much, and thankfully I remembered to do this, that I had to sign up for Storify just to save the tweets Jen and I had received on our wedding day. The tweets are out there, but they're as good as lost if you have no way to relive them.
This isn't an easy problem to solve, especially when you're receiving amounts of data orders of magnitude higher than anything I could comprehend. But there's hope:
@maxvoltar @bryanveloso @mdo No promises yet, they're just hacking. It would still need infra and prod and support behind it. #notsmall
— Doug Bowman (@stop) December 31, 2011
I have faith that they'll eventually solve this dilemma, but eventually doesn't change the fact that you lose a tweet every time you tweet.
Since the day it was released, I've been a happy user of TweetNest by Andy Graulund. I installed it, sucked it up and got my 3200 tweets back in June of 2009. A cron has been working away every 15 minutes since then and I've been happy with having a searchable archive of tweets.
While doing some winter cleaning of one of my hard drives, I ran into a JSON file that was generated by Simon Willison's mytweets script. Those tweets dated back to August of 2008. Excited, I asked Andy if there was any way to get TweetNest to accept JSON. He replied:
@bryanveloso Haven’t made any scripts that do that I’m afraid, but it should be an easy modification in loadtweets if it’s the same format.
— Andy Graulund (@graulund) December 29, 2011
It's been a few years since I've actually tried to write any PHP and whatever I did write sucked, but my mantra for this year has been "Challenge Accepted" so it was time to take it on. The gist that follows is what I came up with. It's a positively horrible hack that successfully gets tweets into TweetNest. I had to do a bit more legwork, since I found out the hard way that Simon's script deleted the user key.
Made TweetNest my bitch. Found an old JSON file, built an upload form for it and now I have 71% of my tweets archived. \m/ #winning
— Bryan Veloso (@bryanveloso) January 12, 2012
I'm posting this here in hopes that the PHP wizards amongst you out there can help me approve upon this, to ultimately submit to Andy as a pull request when the time comes. I mean, I do work for a social coding company. Let's be social my friends! ;)