I noticed a bot crawling my blog calling itself relevantNoise the other day. Curious, I traced it back to its owners. Having looked at relevantNoise’s site, I realised I was just a little miffed but not too sure why. Then I saw this:
relevantNoise is dedicated to mining blogs for business intelligence.
Data mining… That’s why it irritated me immediately. But are they doing anything different to Google, Yahoo et al? Well, yes and no. However, Noel over at Spifftastic thinks they are:
Why should it bother me? It’s a normal part of the internet these days, it’s just something businesses do to gather information for marketing and the like. relevantNoise is no different, they’re just a service that provide this. Well, I suppose the first and most important issue is they seem to treat bloggers (I also don’t like referring to myself as a ‘blogger-’ one of those things I hate about myself) like meat, almost as if we aren’t people. We’re just their way of staying afloat as a business.
We are basically the very core of their livelihood. What we write gives them reason to exist and be paid. They’re a bit like scavengers. relevantNoise simply locates blogs and begins mining them for information. Nevermind that they don’t ask for permission or anything.
I tend to feel a little like Noel. Sure, Google and Technorati spider our blogs and people can find what they are looking for. However, what relevantNoise is doing is much the same as the hotlinkers and spammers; they are freeloading off the bandwidth we pay for and using it to make money. If they want access to any useful business “buzz” from my blog acquired by spidering the site, why shouldn’t they pay me for it? Also, when I do talk about companies, such as last year with the MyDV fiasco, it is not for the benefit of some marketing guru, it is for the benefit of the consumer.
Of course, relevantNoise don’t see themselves in this light. As someone commenting on behalf of the company on Noel’s blog says:
All we do (totally oversimplifying here) is take publicly available data from published blog posts and put it into a very clear and easily-read format for brand managers and similar professionals. So it’s all stuff they could pull themselves out of blogs—we just do it much faster and more effectively than they could, and analyze it better and faster.
Fine; this is perfectly true, it is publicly available data. However, it is also my work and my bandwidth, so let ’em do it the hard way or cough up the spondoolicks.
Willmacc over at A Daily Rant is less forgiving than either I or Noel:
Mining Blogs for business intelligence? Sounds a lot like plunder and pillage to me.
Willmacc has drafted a .htaccess entry to block relevantNoise’s bot.
deny from 64.21.0.0/17
deny from 64.21.128.0/18
#user-agent deny
SetEnvIfNoCase User-Agent “relevantnoise” bad_bot
Curiously enough, it found its way onto my .htaccess file. Wonder how that happened…? Of course, if the people over at relevantNoise want to make me a suitable offer for my business “buzz” I might consider removing it. I won’t stop up waiting, though.
Update: Following on from the comments, it is worth pointing out that relevantNoise are in the business of reputation management. Public outrage risk is something modern businesses have to be aware of and prepare for. However, relevantNoise seem to have got off on the wrong foot as Aimee commenting here on behalf of relevantNoise lets slip:
We’re getting a fair amount of flack lately for what we do, so it’s nice to see a post in which we’re not called “spies” or something similar.
Joseph Dolson, responds:
Indeed, a lot of people don’t understand the whole concept of mining public data for information: like you say, the information is already out there, you’re just searching it out and analyzing it.
I, and others, understand rather better than either Aimee or Joseph give us credit for. As Noel points out on his own blog and here in the comments, it is the casual, almost cavalier approach to the bread and butter providers that causes annoyance; we are merely product to be harvested and we are expected to contribute financially to a company that then sells our output when we provide it for free. Perhaps the most amusing aspect of all of this is that for a company that is dedicated to building a business in reputation management, the people at relevantNoise seem to have been rather careless with their own.
Surely this should be supported. I mean, if you are complaining about a company on your blog, surely you want them to know about it so that the issue can be resolved? Otherwise, you might as well tell your tale of woe to a stool in the corner.
The world is imperfect, and there’s not much to be done about much of it. In complaining about this, are you not suffereing from a similar blindness to the Blair/Brown Government, to name but one?
Best regards
I don’t have any particular issue with what they are trying to do; rather, the manner in which they are doing it – it is impolite to use someone’s bandwidth for free and then effectively sell the results. If they made their results freely available as do Google or Technorati, or, as I suggested, bought the bandwidth, you wouldn’t hear a peep from me.
If I have a complaint about any company, be assured, they will know all about it.
No. I don’t wish to be spidered by this company, so I block it, just as I block others who would take similar advantage. I’m not sure “complain” is the right word. I disapprove of their business model and I’m voicing that disapproval – and why not?
Dr. V: The issue is not that the company will hear about the complaint, it’s the manner in which they acquire information about the complaint. It would not be an issue if the company itself found your blog entry and dealt with it individually.
Instead, however, the company hires relevantNoise, who get paid to take up our bandwidth by running their spider across websites without any concern for whether or not the bandwidth used will actually be made worthwhile to the people their spider affects. Essentially, relevantNoise requires bloggers to accept their existence and tolerate their use of our bandwidth in order to make their money.
Do we get a dime for their work? Nope. Do we receive any thanks for allowing them to survive? Certainly not. As I’ve said before, all we are to them are pretty much the equivalent of livestock.
Worst of all is that you cannot opt out of this. relevantNoise does not tell you of any way to block their bot, they do not mention the specifications of their bot and how much bandwidth it uses and whether or not it shows restraint so the website it is spidering does not get hampered by its activity. There is also no indication as to whether or not it pays attention to robots.txt.
It’s a bit like being told to sit down, shut up, and take it. So, in the end, I decided to block them.
As far as I’m concerned, relevantNoise can rot in business hell. At the very least they won’t be acquiring anything off of my site without doing it by hand, and even then I’d probably ban them if I found out.
I’m wondering if you have considered use of robots.txt. This is a way of telling robots the parts of your website (or even all of it) that you do not want to be downloaded, or are not worth repetitively downloading; it can be set selectively for particular robots (rather than for IP addesses). So it is more targetted than a blanket ban using .htaccess or a firewall. It’s a mutually cooperative mechanism, so can be viewed as a firm thought polite request.
Of course, a robot that ignores such a firm but polite request does, perhaps, deserve stronger action; perhaps even stronger action than you have taken.
Best regards
Some don’t understand, others just disagree. I can certainly understand why some people would take offense at the way relevantNoise works – like you say, they aren’t paying you for it, they’re just taking your bandwidth for their own gain.
But personally I don’t have any problem with it – the bandwidth they use is insignificant, and I didn’t have any particular intention of profiting from my blog posts in the first place.
It’s my opinion that companies like relevantNoise are predominantly selling their labor: the time they’ve spent developing their software and whatever management they need to do in order to provide reputation management services.
I don’t think I’d hire them, however – there are much easier ways to track your presence in the so-called ‘blogosphere’ – Google Alerts, for one.
I don’t have a problem with the principle; it’s the cavalier attitude that goes with it that raises the hackles. I don’t seek to make a profit from my blog, so I’m not too impressed when someone else seeks to do so at my cost (however negligible) and without so much as a by your leave.
As I mentioned in my update, for a company that seeks to sell reputation management, they have a learning curve ahead of them. Physician heal thyself, it would appear… 😐
Hi, Longrider — I respect your opinions, and I understand and (again) respect that you’ve blocked us. I’m totally cool with that. But I did want to set the record straight on one point: I’ve just spoken with our developers and they’ve assured me that we do, in fact, use minimal bandwidth. Our bot will visit your site once per update. So if you update once a week, we’ll take the same bandwidth as any single visitor reading your current post.
Thanks to you all for the food for thought! I appreciate the comments from those who get what we’re doing (essentially handing your opinions about certain brands/candidates/products/services to their respective managers) but I completely hear and respect the opinions of those of you who don’t.
Longrider, we’ll work on the “cavalier attitude.” 😀 (It’s all me — everyone else here is far less extraverted.)
Best of luck to you in 2007!
Thanks for your comment.
The Internet has developed an etiquette all of its own. Sometimes it is not easy to determine exactly what it is until you break it. This is one of those occasions. What you do is subtly different to what the search engines do. To claim otherwise is being at best, misleading and at worst, disingenuous.
It would be nice to see something on relevantNoise’s home page that recognises that what you are harvesting is the work of others who provide this for nothing. Some recognition and a nod of gratitude would be a start. It’s called saying “thankyou”. After all, without blog writers, you don’t have a business model.
So, yes, I do understand reputation management and the effects of public outrage risk – I come from a business background that has had to deal with the spectacular fallout caused by it.
Duly noted. I would counter with the fact that (as doctorvee pointed out) most people complaining or praising a particular entity and publishing (and syndicating!) that opinion to their blog fully expect those opinions to be read, and would probably be pleased to know that their work (which we’re “harvesting”) has actually reached the appropriate ears.
But, as I’ve said, I hear what you’re saying, and I’ll take it under advisement. I’m a blogger too, with a few personal blogs out there, and I can see where you’re coming from.
Rather a lot of, I have to say, very surprising outraged proletarian type rhetoric here Longrider, which I completely disagree with. The complaint that precious bandwith and server resources are being eaten up by a few requests for static content every now and again is groundless and petty. Libertarians have business ethics too it seems. Most blog writers will not even know this is happening and since they use free software and (not in your case) free hosting to make the world aware of their opinions they have no case to complain about what happens to their data. If you don’t like your blog being in the public domain, and are not willing to accept the implications of this (which go beyond internet codes) then either a) don’t do it or b) add security to your blog so that it is not freely available to anyone. Although google do indeed offer their services free, they make vast amounts of money by indexing and attempting to categorise (amongst other things) blog rants.
Really? Maybe that’s because I’m a member of the proletariat, eh?
Steven, you are free to disagree as you please. However, it changes my position not one whit. This is because you have missed my point.
I have no objection to people making money by sifting publicly available information and selling the results. Although the term data mining has connotations that immediately annoy me. It’s a poor choice of wording, I feel. My objection to the bandwidth issue is not petty, it is relevant to the underlying issue; one of attitude. The hotlinkers over at B3ta were not using a huge amount of bandwidth, but I still put a stop to their antics on principle. It’s not the quantity, it’s the principle. If you are going to use someone’s bandwidth in a way that is not the usual accepted one then the least you can do is ask or make some effort to offer an explanation. This is very much the same as the Blogshares discussion a year or so back. They tried the “google do it, so we don’t have to bother” argument and that didn’t wash its face either.
When I visited relevantNoise’s site, my hackles raised. Why was that? it was because I felt exactly as Noel; bloggers were being talked about in a manner more appropriate to livestock than people who are central to the business’ success.
It would be a simple matter to deal with what was entirely foreseeable beforehand by putting a FAQ page for bloggers that explained why relevantNoise exists, what it is trying to achieve and why bloggers are so important to its success. In it, they could include the answers that Aimee has been busy trotting around various blogs providing in her attempts to regain the initiative. Such things as the details of the bot, bandwidth use and, importantly, for those who do not wish to take part, an opt out. Most businesses have to sell to their clients. One such as this, based upon using someone else’s output for free also has to sell to its suppliers. This, they failed to do and the result is discussions such as this one. It could have been avoided so easily. In all probability I would have still blocked them, but I wouldn’t have felt inclined to voice my disapproval.
What I am asking for here, in one word is “courtesy”. Is that really too much?
———————-
One final point – as people have mentioned the comment by Doctor Vee. If I have a problem with a company, I will not blog about it and hope they get to know that I’m not happy. I will contact them directly and leave them under no illusion about how I feel. If I subsequently mention it here, it is entirely for the benefit of other consumers to help them make decisions about their purchases. If the company wishes to respond then that’s fine. However, they are not the primary target audience and my words are not intended for their consumption.
I’ve decided to unblock relevantNoise’s bot for the time being, since I’m expecting Aimee’s work to produce changes to the website that fix the issues you and others have mentioned. That is, I want information to be present that shows that we are important not just to the people they’re selling to but relevantNoise as well, and including information on the bot and how to block it if we decide to do so.
So, in regards to relevantNoise: don’t treat us like meat and we won’t treat you like rats. You give us what we ask for and I’m sure a lot of people will be more than happy to let your bot and team do their thing.
Noel, I’ll keep an eye on things as I’m hoping that Aimee’s comments here will lead to a suitable change to their site. Like you, I’ll reconsider should they do the decent thing.
You know, for a while, they never had their user-agent in their hits, Then, no respect of robots.txt – Umm,,, that’s when they got the boot. “It Ain’t Baseball – 3 strike rule” – more like “Gone in 60 seconds”.
I’m less forgiving because I see it 24/7-365…
Also, this and about 100 more do the same thing and hundreds more we don’t know about.
Two points: 1) you’re a blogger and, just by nature of the term you want your opinion heard by as many people as possible. Otherwise you’d put it in an access protected blog so that only your friends can see it. 2) This is not a new phenonenon. Everyone is taught from a very early age that information is generally availble to the world – provided it is not copyright protected. For example, I can freely read through 1000 books and make note of every authors opinion on a subject. I cannot video record a movie in a theater. What RelevantNoise is doing is marking down the opinions of bloggers (positive and negative) and making note of it.
Your points have been covered pretty extensively in both the original article and the subsequent comments. However, I will reiterate; no one is objecting to anyone looking for information and noting opinions. What rankles is the manner in which the company chooses to do it. Yes, people are free to look at what I write and note my opinion. I am equally free to withhold my cooperation with their methods if I object to their behaviour. This is what is happening here.
On the matter of copyright – all blog postings are subject to copyright law.