First some thank yous. I’ve tried to quote everyone who wanted to be quoted, and to all those who gave me info anonymously, you know who you are and thanks.

The iTunes charts are very important for the Music Industry, but the podcast charts are a Sudoku puzzle wrapped in Rubik’s Cube. Although knowing the formula (or its approximation) wont help you usurp Ricky Gervais or Chris Moyles but it my help you; optimize, maximize and …err caramelize.

My attempts over the last few weeks to reverse engineer the algorithm have exposed 5 areas that I feel are the most important in a successful podcast. The data I have used is a mixture of empirical facts, cold hard statistics (lovingly donated by the some of the top podcasters) and gut feelings on how I’d create a ranking algorithm (having spent years working on ranking systems in a vain attempt to topple AltaVista, Yahoo! and Google, in that order).

I’ve also collected plenty of anecdotal evidence to back up my clams and help me hypotheses about the exact “mix of ether and witchcraft” Apple use.

Subscriptions & Downloads

Everyone I spoke to agreed the most important element in climbing up the charts in iTunes are subscription. However this is where the consensus ended, some place almost all their faith in subscriptions others plumbed for a weighting between subscriptions and downloads.

My personal view is it more likely to be a weighting, I’ve heard suggestion of 60:40 to 80:20 in favor of subscriptions. This means for every subscription, you have to get between 60-80 downloads to match it. I like this model because it awards consistent podcasts and those with a large subscribed base.

My evidence for this can been seen for yourself, you’ll notice an increase in chart position for podcasts that have just released a new show. With popular podcasts this might only translate into a place or two, and for podcasts in the top 10 the effect may not even be visible, but if you look towards the bottom of the cart you notice a lot more movement.

Of course this has it issues, and has lead to the Gervais Effect (I just made that up) this is where a podcast, such as Ricky Gervais’, has a massive subscribed base and a 5 minute podcast every 6 weeks is enough to keep him in the top 10.

Plays

During the investigation I heard a rumour that “plays” we used as a metric. I am 99.99999% sure this in not true, for a start people would know if Apple were eves dropping on your listening habits plus it would require Apple to download every podcast and analyze it to determine it uniqueness (and for iTunes to also do this). I don’t want to get too technical but, rest assured Apple don’t track you listening habits (unless you use the Genius feature.)

Reviews & Editorial

This is the most contentious area. Nobody was willing to speak, “on the record ” but I did dig up some great nuggets of information. Apple (as you would expect) likes to keep a tight control on all aspects of their business, iTunes is no exception and neither are the charts. Apple has a full veto over any and all podcasts but also can apply some of it own weighing, both positively and negatively. This is most noticeable with the BBC podcasts. They regularly defy the normal downloads rules and post higher chart entries than “independent” podcasts.

While reviews on their own won’t get you higher in the charts they will get you in “New and Notable” and possible as a “Featured” podcast. I am reliably informed if you make it as a ”Featured” podcast you’ll be contacted by Apple and if they like you, maybe even buy you lunch.

Video vs Audio

This only comes into play in the combined charts. A simple glance at that chart will tell you, its about a 50:50 split. But there is further evidence Apple prefer to push video over audio.

The iTunes Clepsydra

1. Subscriptions and Downloads (possibly reviews and ratings).

2. Read off your rank and compare with other Podcasts to determined chart position. Apple favored Podcasts (BBC et al.) might have a different scale, thus artificially inflating their position.

3. Old subscriptions and downloads “drip” away, this method favors regular podcasts with large subscription base.

4. Archived download data, possibly used to determine ”Featured” and “New and Notable” podcasts.

Improvements

Working for bands, I would often bump up against the UK Chart Company Rules [PDF] a vast and seemingly random collection for things you can and can’t do when releasing a single. Things like track length, price etc.

Having researched the iTunes chart I can now see why these rules are necessary. Is it fair that The Onion at < 1 minute (but daily) be ranked along side something like Answer Me This, weekly and 30x longer? How about Ricky Gervais, intermittent and only ever adverts or recycled material. Compare this to Stephen Fry, again intermittent but superbly produced and always original.

One thing is clear Apple needs better filtering, to enable user to find new stuff. It should publish downloads stats and allow podcast producers to access analytics on their shows. Finally a podcast that has not been updated in the last 3 months should not feature in the top 100, this would churn the chart and allow fresh content to reach new audiences.

Apple has a wealth original content (some of it much better than the latest teen drama form ABC) that it largely ignores, it could better market it and help monetize content, but that my friends is a topic for another blog post.

Thanks to Scott and Matt from the Guardian for you invaluable insights.

The recent Government announcement that they were considering monitoring Facebook, MySpace and other social networks for dubious behavior passed by under the radar.

None of the fervor of ID cards, when it’s blatantly more of a infringement of personal freedom.

The problem with any law like this is it doesn’t effect the average voter, this is aimed squarely at (rogue) teenagers (and early adopters). By the the time they can do anything about it, it will have been on the statute books for years.

With the recent announcements on thought controlled devices how long will it be before the government is recording our thoughts under the guise of theses laws.

I’m not too worried about Jackie Smith knowing how many times I check Twitter ,but I’ve got a real paranoia about her recording my thoughts. Just think (excuse the pun) how many times have you thought something that if you said it, you would be arrested?

Paranoid or concerned citizen? Which are you?

I’ve had this idea for a few weeks and I finally want to move it forward or forget it.

In a nut shell Prior Art will be a podcast (weekly or bi-monthly to start) that will discuss ideas and inventions sent in by listeners. The stated invention is to enter into the public domain ideas and systems before large corporations and patent trolls can patent them.

By discussing them “on air” with technology experts and providing a forum to discuss these ideas I hope to speed up the dissemination of great ideas ultimately leading to real world and practical applications.

The show will be 30-40 minutes, consisting of 3 or 4 segments with technology experts, commentators, bloggers, developers etc. Interviews will be conducted over Skype, interviewees will be encouraged to constructively comment on the ideas of that weeks show. A range of experts will discuss the technological and business aspects of the idea.

Listeners will be encouraged to comment on the ideas and also create examples, hopefully leading to some interesting collaborations and great new technology.

A tentative tag line is, “Open Source Ideas” and the show its self will be licensed under creative commons.

What I need is, firstly you thoughts on the idea and show format. Secondly ideas to discuss in the show and thirdly experts, commentators bloggers, enthusiasts willing to give up 30 minutes to recored their segment. I won’t be paying anyone, but you can shamelessly plug your blog, book, web site etc…

How can you help?

1. Send me ideas

2. Sign Up to record a segment (send me an email or get me on Twitter)

3. Tweet about it (don’t forget a link to this blog post)

4. Leave a comment bellow about ideas etc

5. Digg this blog post

6. Blog about the idea.

Local news just got that little easier to find on Twitter. The micro-blogging site has exploded over the last 2 months and with it a seemingly inexhaustible supply of information.

Below is a list of local news services on Twitter… enjoy.

You may have already read my how-to on integrating Twitter and Growl but this is the upgrade! I will go through the code for those interested, please help me improve the code by leaving a comment, good or bad.

I have some quite ambitious plans for this project, but this is not the time to bore you with promises, so vaporware it will remain – for now.

I’d like to start by thanking everyone who has commented on this project so far – it’s been such a big help, keep them coming.

What’s new?

If you can’t see a feature you requested its not because I am ignoring it, it’s simply I haven’t got round to it or it been superseded. As alway comments will always be read so you can always bug me there.

1. Profile images now supported

2. Run in the background and optionally use launchd

3. Sticky notifications for selected users

4. Loads more command line options

Bugs Fixed

UTF-8 encoding issue.

Download

Download here. Remember you’ll need some command line experience. See an example launchd plist file here.

Install

Installation is relatively simple, you will need to install some Perl modules, refer to the previous post for details.

Running

NB: I have split this command for ease of readability.

$ ./trowel -u <<EMAIL>> -p <<PASSOWRD>> -vv \

-I guykawasaki,twitter,stephenfry,rjstelling,macrumors,2009show \

-t twitter,stephenfry,rjstelling

-I tells the script what to notify you about

-t makes the notification sticky

Know Issues

There does seem to be an issues when running in launcd mode where Tweets are duplicated, any testing and info you can supply on this bug will help no end.

Documentation 

NAME
		trowel - display Twitter messages with Growl.

SYSOPSIS
		trowel -u -p [options]

EXAMPLES
		trowel -u <username> -p <password> -i 5 -l 180 -f "%t %d" \

		-t twitter,stephenfry,rjstelling \

		-I guykawasaki,twitter,stephenfry,rjstelling

OPTIONS
		-u, --username
			Twitter username

		-p, --password
			Twitter passord, if omitted it will be 

		 	requested interactively

		-i
			interval between displaying tweets

		-l, --poll
			time between polls of Twitter feed

		-x, --exclude
			list of users to exclude

		-t, --sticky
			list of users who's tweets are sticky, 

			-x and -i will override this

		-I, --include
			list of users to include

		-o
			output to STDOUT only, by-passing Growl,

			 use this for piping to another application

		-n
			initial number of Tweets to request, default is 5 

		-f "<format>"
			format of the Tweet
				%u - user
				%t - tweet
				%d - date time
				%l - location

		-g, --config
			a configuration file that sets command line 

			parameters, this function is not implemented

		-h, --help
			show this help 

		-v
			verbose mode

		-vv
			very verbose mode

		-vvv
			debug verbose mode

Passwords are the necessary evil of modern day computing. Just try and think how many passwords your have; Hotmail, Facebook, Amazon, Blackberry PIN, ATM PIN… the list goes on and on. 

Our online worlds would cease to function if Firefox or Keychain or whatever windows uses forgot all our passwords. Thats why so many people use the same password, for everything! And online bad guys know this, once they’ve hod-winked you into handing over your Hotmail password they can get into your Amazon, your Facebook gathering mothers maiden names, pet names, towns and cities of birth. All the info they need to steal your identity. 

Luckily the banks are hip to this and don’t let you choose your own password, a series of codes, tests and randomly generated numbers are required to log in. But this is all rendered useless if the thief walks into your local branch armed with a copy of your birth certificate and 3 months of utility bills.

What’s wrong with my password you ask? Well as my Mum recently discovered using a dictionary word for her Hotmail, it was just a matter of time. And a random string of letters isn’t much good either, with networks and processors getting fast enough to make brute force attacks feasible.

Lets do some Maths! Take your normal 8 letter password, all in lower case and no numbers that gives:

n(r) = 26(8) = 208,827,064,576

Now if we add in upper case:

52(8) = 53,459,728,531,456

And now the gold standard, numbers and punctuation , . – _ ( ) [ ] \ / ? < > ! @ $ % ^ & * an extra 20 possible characters:

(52 + 10 + 20)(8) = 2,044,140,858,654,980

Now this is getting close, even Roadrunner would have trouble cracking this, but it’s still not enough. It wont be long before we have this type of processing power on our desks and on our laps and eventually in our pockets, so we better come up with a solution.

The Dynamic password, a cypher with an algorithmic portion.

The user would pick a the normal password, mixed case, numbers etc. but also a function, these functions would take values such as, time, date, weekday etc and produce an output that would be appended or prepended to the password. This would make passwords only valid for a few hours or days at most. And vastly decrease the odds of a brute force attack.

Examples

<Cypher><Yesterday’s date * your birthday> 

<Tomorrows Date><Cypher><The hour * today’s date + 1)

This ‘new’ passwords have the advantage of automatically changing every day, week, hour or even minute. This means if a hacker does crack your password, it’s no good in 60 seconds. Or if it only changes every day, he’d have to have a computer fast enough to check all possibilities in 24 hours other wise start again.

Don’t forget to express yourself in the comments.

This is the original post is started to write when I went on a rant about Google Search killers, sorry about that. Here is a more succinct post abut what I’d like Google to provide.

Power Search

Let me know in the comments of power search features you’d like, but 2 that would be useful for me would be; ‘ip4:’ and ‘ip6:’. A query of:

ip4:17.149.160.49

This would return results form this IP address 17.149.160.49 (’ip6:’ is the IPv6 equivalent).

Spell Checker

As you might of guessed my speling sux. It’s a combination poor typing skills and dyslexia. So the thing I love most about computers is the spell checker. Apple make my life easier too by providing a build in spell checker for any Cocoa application. But by far an away the best spell checker I have ever used is Google, so much so that I will often find at the end of a day 10 or so window open with search for; ‘avaibule’, ’sussinct’ or ‘priverlage’.

A handy widget or API for this would make me happier than a gay dyslexic at a pianist convention.

Upgrades

There is a lot of technology in Google that doesn’t quite work. Take the ‘related:’ command, supposedly it list pages and site that are, well, related to a site you specify. It only works for a handful of sites and then with curious results.

I recently had an discussion* with a colleague about why Google rules the internet. We were having our bi-monthly discussion (like all start ups do) about how we’d love to launch a search company; “we’d do it some much better”, “our interface will be way cool”, “think of the money”.

“A little top down thinking about crazy projects is light relief in these hard times and I actively encourage it, my secret hope it it might plant the seed of a future project the will wipe out my student loan once and for all.

My argument was along the lines of, 1. Anyone can collect data and store it for search and 2. What makes Google the best is the way the molest your query and give you what you want not what you asked for. This is why when you type in “june sarpong” you don’t get any calendars or things happening in June. This, my argument continued was the USP the thing no one else did as well or at all.

“What You Want, Not What You Think You Want should be Google’s new mantra they could shorten it to they snappy WYWNWYTYW, say it with me, W, Y, W, N, W, Y, T, Y, W!

You’d think with Microsoft desperate to get any sort of traction with their online real-estate, Yahoo! Fucked! they would e trying hard to out do Google in the search space. This week however I was checking the server logs (as I do every Monday morning) and I noticed something strange, Google as usual was the Number 1 spider on our servers but second was Cuil! Imagine my surprise.

“Figures from Net Applications show that Cuil had a market share of less than 0.01 per cent in October.

Microsoft and Yahoo! the respective Number Twos in online search aren’t even out-indexing a poxy start up like Cuil. You might not be able to compete on search power but at least have the courtesy to provide you dwindling user base with up-to-date results!

“Google’s profits were up 17% to $1.21bn (£608m) for the three months to the end of December, 2007.

THIS POST HAS BEEN SUPERSEDED BY Twitter + Growl = Trowel v1.0

OK, so I’m new to Twitter. I didn’t (don’t) get it. Who wants to know what I’m doing? Well as I keep telling myself thats not the point. The point is to spread information, I had a lot of fun on Election night tweeting away about the coverage and calling states (I was surprisingly accurate).

The one thing that did annoy me was it added another website to check – not just a lunch but a refresh every 5 minutes. Something had to be done.

For anyone who has Adium installed on their Mac, you will be aware of Growl it’s a notification system that has yet to reach it’s full potential. As with background messages I wanted my incoming Tweets to be shown in small unobtrusive transparent bubble.

To cut a long story short – I couldn’t find anything. I found a few Ruby scripts (that didn’t work), I tried some nice applications but I didn’t want to run another app I wanted Growl!

So an hour or 2 or Googling and research for scripts I found some Perl modules that I could combine to produce the desired output.

My steps and code are listed below, its not 100% fool proof – please leave comments on how I can improve it.

1. Installing the Modules

You will need only 3 modules Mac::Growl, Net::Twitter and Date::Format, if your lucky Date::Format might even be already installed. Type the following commands into terminal.

$ sudo perl -MCPAN -e shell

This will launch a CPAN command line session to install the modules type (at the prompt):

> install Mac::Growl

Agree to any dependancies, when it done type:

> install Net::Twitter

Again agree to all dependancies, and lastly:

> install Date::Format

> exit

If you had any problems it may be because You need Apple’s Development tools installed, these are a free download form Apple. Please leave any problems in the comments.

2. Coding

Now for the code, just copy the following into a empty text file and save as some thing sensible like growl_tweet.pl:

#! /usr/bin/perl -wT

use Mac::Growl ':all';
use Net::Twitter;
use Date::Format;

my $app    = 'Growl/Twitter';
my @names  = ('New Tweet');

RegisterNotifications($app, \@names, [$names[0]]);

my $tweet = Net::Twitter->new( username=>'##REPLACE WITH USERNAME##', password=>'##REPLACE WITH PASSWORD##' );

my $last_id = undef;
my $switch = 0;

while(1)
{
 my @tt = ();

 if($last_id)
 {
 @tt = $tweet->friends_timeline({count => 200, since_id => $last_id });
 }
 else
 {
 @tt = $tweet->friends_timeline({count => 5 });
 }

 foreach my $t (@{$tt[0]})
 {
 if(!$switch) { $last_id = $t->{id}; $switch++; };

 PostNotification($app, $names[0], $t->{user}{screen_name}, $t->{text});

 printf("%s: %s\n", $t->{user}{screen_name}, $t->{text});

 print "sleep 4\n";

 sleep(4);
 }

 $switch = 0;

 print "sleep 37\n";
 sleep(37);
}

exit;

3. Running

Now, simply run the script:

$ perl ./growl_tweet.pl

And bingo! All your tweets appear! Leave a comment if it works and especially if it doesn’t – if there is enough interest I might consider adding features and real installer!

4. Here’s one I made earlier

If you would like the code, you can down load it here (with out any formatting issues).

http://www.echo-technology.co.uk/development/twitter+growl/growl.pl.txt

This website was created to help people who care about local issues. Whether you need access to useful information, want to highlight a particular topic to other people, discuss ideas with like minded individuals or just show your support and keep up to date with current affairs in your area, Bristol Matters is the place for you.

http://www.bristol-matters.co.uk/