Ever since the launch of iPad in March 2010, it has redefined the way people consume information. A trend that started with the iPad is now being carried forward by other tablets that have entered the game, including the very recent Kindle Fire. How big is the tablet market ? Has the surge of tablets impacted consumption of News ? What are the different News apps out there and how do they stack up against each other ? What can be expected of News apps in the future ? We attempt to answer some of these questions here.

The iPad, which leads the tablet space by a huge margin, has already sold over 35 million tablets – and that is excluding the numbers for the December quarter of 2011. Kindle Fire too has sold over 3 million since its release. The other Android tablets are yet to catch up. A conservative estimate of all the tablets shipped till date falls anywhere between 40 and 45 million. What’s more important is the pace at which this market is growing. Estimates suggest that iPad alone will sell close to 68 million units in the year 2012. This number is expected to be around 138 million in the year 2015, out of the 294 million tablets expected to sell that year as per Gartner’s report . Clearly, this is a market that Media houses, Developers and Marketers cannot overlook.

The Media Tablet market

But, what are these 40 million users doing with their tablets ? A recent study by conducted by Pew Research Center, in collaboration with The Economist Group, throws light on the media consumption patterns on tablets. News consumption very nearly equals Email as the activity of choice on the tablet. Close to 53 % users get news on their tablet every day (54 % for email). And 77 %, or 30 million users get news on their tablets atleast once a week (“News Users”).

Tablet Usage

These numbers give news companies a lot to cheer about. Big media companies and a bunch of startups are all competing with each other to bring these users to their news apps or sites. Close to two thirds of the tablet news users, that is 20 million users, already have one to three news apps installed on their tablets. While tablet news is a trend that is here to stay, the details deserve a look.

Though tablets are replacing PCs as a medium to access news, the browser still remains the more popular means of consuming news. Only 21% (6.3 million) of the tablet news users prefer apps over the browser to access news. And, even here, preference remains deeply rooted for apps published by News companies rather than aggregators.

Access to News

Now for the big question : Why is the mindshare of aggregator news apps so abysmal ? What should aggregator news apps do to improve this ?

Before we attempt to answer this question, let’s take a look at the current set of aggregator news apps in the market and where they fall short. The ecosystem of aggregator news apps can be broadly classified into two : Personalization driven and Discovery driven.

Personalization driven : These apps allow the readers to consume the content of one or more web sources. A list of web sources is made available by the app and readers can choose the ones they prefer to view from within the app – hence the word personalization. Readers can also view their Twitter and Facebook streams from within the app. There is little or no curation of content. The mainstay of such apps is their usability. The list below shows Macworld’s ratings for a few such apps.

Personalization Apps

Discovery driven : These apps provide readers with channels of curated articles for various topics. Very often, these articles are drawn from sources that readers might not be aware of. The channels and articles there in are curated based on the choice of topics by the reader. The readers usually cannot change content sources within a channel. the list below shows Macworld’s ratings for a few such apps.

Discovery Apps

While a detailed review of all the above apps is out of the scope of this article, there are a few important things to note.

Most common features :

Here are some features you might find as a part of every app (barring a few exceptions)..

• Content Aggregation : Content from multiple sources, either given to the user as clusters or as individual feeds.
• Social feed integration : Facebook and Twitter can be accessed from within the app.
• Basic customization : Subscribe new content sources or to new channels.
• Share content : Articles can be shared through Twitter, Facebook or Email

Differentiation factors :

Here are features that apps try to differentiate on..

• User Experience : Flipboard sets the standards here. Publishers craft their pages as per Flipboard’s needs.
• Layout customization : Readers can alter the look and feel as per their needs. The options could include number of columns, simple list view or summary of each article etc.
• Breadth of content sources : YourVersion HD is the only app that integrates Quora. Flipboard now gives Instagram as one of the options.
• Interest based curation : The interests of the reader can be mined from his Facebook or twitter profiles, from the articles he reads or from giving a thumbs up or thumbs down on specific articles. Zite allows readers to request more content from a particular source or more on a particular topic.

The short falls :

Here is a list of what you might not like about some of these apps (as per this and this)..

• Pulp : The app does not provide an option to view your twitter and Facebook feeds.
• Taptu : The home page appears crammed, with 56 visual elements vying for your attention.
• News.me : This app is essentially a paid twitter reader with no other feeds available.
• Trove : The app provides an inconsistent reading experience because of the in-app browser.
• Flud : The app forces feeds into categories. The user interface is non-intuitive.
• Hitpad : Content is presented based on trending stories, making particular topics hard to find.

The leader board :

Not all apps are created equal. Two apps, in particular, have caught the attention of readers – Flipboard and Zite. Flipboard has been downloaded over 4.5 million times. Here is how they compare :

Flipboard and Zite

And now for the wish list :

So, there is a glut of apps, some fall short of expectations and some do better, but the browser and apps made by news companies remain the go-to destinations when it comes to consuming news on tablets. It’s now time to think about what a highly engaging aggregator app should provide, if we wish to reverse the above trend. Beyond the common features, here is what I think these apps should wow us on. Though this is my personal wish list, I am fairly sure that a large set of readers will vote for this.

• UI : The UI has to be extremely intuitive, simple and elegant without being loud and in the face. Anything that takes the attention away from reading is a no-no. There should be consistency in how every article opens inside the app. A case where some articles open in a reader and some in a browser is clearly despicable. The UI should also allow for few minor customizations, like font, text colour, number of columns, list and summary view etc.
• Customization : I prefer subscribing to topics and allowing the app to choose the best available content sources for each topic. That said, I should be able to add or remove specific sources under each topic at my will. This will help me discover interesting content from sources that were previously not known to me. It also means that the app should provide a sizeable list of content sources on various topics to cater to my manual addition needs.
• Interest learning : The initial set of topics relevant to me can be bootstrapped by mining my Twitter, Facebook or Google Plus profiles. This can be particularly useful if I am a heavy user of any of these and it can save me significant time in customizing the app.
  
  The app should continuously learn about my interests – both implicitly and explicitly – and give me better content. Implicit learning should be gathered from the kind of articles I open, the amount of time I spend reading these articles, the sentiment expressed in comments and so on. Explicit learning can be gathered when I give a thumbs up/ thumbs down on articles, ask for more content on particular topics and so on. Instant gratification is extremely important when I provide explicit feedback. I need to know what the system has learnt about me with each input I give, else I might not be motivated to continue giving information. As an example : If I give a thumbs up on an article, the app should tell me : ”We learn that you like these topics : ….”.
  
  Apps are still silos when compared to the browser. The browser, with its support for cross linking, helps us enjoy the web in all its glory. If an app needs to challenge this, it should learn all my interests over time and bring into the app all the information I might want to read about, highlight all the links that might interest me and bring in the target pages into the app as well – in short, the app should be “THE” place for all the news I need, never having to look elsewhere for anything. Thanks to “App fatigue”, I have now lost the will to go to multiple apps for the news I need.
• Social feel : The usage of social profiles in apps today is mainly restricted to providing access to these feeds from the app, allowing readers to share articles over these networks and in the more advanced apps like Zite, mine out interests to tailor content.
  
  But the consumption of content still lacks the social feel. What I mean by this is that the app should match readers interests and suggest people with similar interests with whom I can share articles. Sharing an article with my entire network means adding a lot of noise to the system – an article that is relevant for a few might be spam for many. The app should also provide an interface where I can access all the articles that have been shared with me.
  
  Commenting lacks personality. I should be able to view opinions and insights about every article under the title “What your friends just said..” rather than just “Comments”. This implies that these inputs should be coming from those whom I know.

Let’s hope that app developers sit up and take notice. Let’s hope that apps truly take us beyond the browser experience. Let’s hope I do not have to open 8 apps to get to all the information I need. Let’s hope serendipity prevails.

Many bloggers have been in panic over the last few months, since Google has been involved in a huge clampdown – banning AdSense on sites that have been part of traffic exchanges. At the outset, traffic exchanges have not been doing anything illegal – they are promising readers to move around the web without getting back to a channel (Google, Twitter, or Facebook). However the interplay of human behavior, along with the model traffic exchanges have created is creating big hurdles for Google – both in ensuring value to its advertisers and ensuring efficiencies of their systems. This article is an attempt to unravel this exciting network phenomenon. Read on.

The AdSense policy on Traffic Exchanges

Way back in 2007, Google Adsense blog came up with post recommending people not to get onto traffic exchanges. The post clearly suggests that while there may be a value in being part of them, they “may” cause invalid clicks. Given that many bloggers depend on Google solely for their livelihood – there is mass fear about adopting high quality products that help improve user engagement on their sites.

The resurgence of traffic exchanges

Ajax and the ease of installation of widgets has caused a resurgence of traffic exchanges. Traffic exchanges provide value – both to the publisher and the reader, in getting contextual and new unique users from other sites. Sites like MGID, Wahoha, 2leap have been growing leaps and bounds. Testimonials of people saying their traffic increased by 500% in a week makes one wonder if this is a big ponzi scheme. To understand this, we should understand how traffic exchanges work.

1) You lease out real estate on your articles to the TE (Traffic Exchange) widget. The widget decides to show outgoing links here dynamically.

2) When users click on the TE widgets articles on your page, the TE remembers your site (or a set of pre-selected pages), and promotes them more on other sites.

3) You get back traffic – often much more than what you sent out. Makes you wonder how this can be a non-zero sum game. The trick is that every click on the widget takes a user to an intermediate page – where the user is link baited to click on more than one article. They all open in new windows in the background. One click turned to many. Non zero sum.

Due to the overwhelming importance given to user click behavior, links that get promoted mostly look like this. I know you cannot resist clicking on them. Unfortunately, these are not linked to.

The typical nature of recommendations on a traffic exchange widget

Traffic exchanges promise return traffic to be proportional to the outgoing traffic from a publisher. So a publisher is incentivized to click on many of these links himself – in expectation to get back unique visitors.

Google and click fraud

Ever since Google introduced their Pay Per Click program for publishers, publishers have been trying ways to increase their revenues. Long tail bloggers and publishers have been using several ways to click on ads on their sites.

• Click on an ad on your own article 3 times a day.
• Invite friends to click on ads on your site 3 times a day.
• Get into an agreement with other publishers to click on each other’s ads
• and so on

Google’s value proposition to the marketers gets diluted with bad clicks. Since a long time, Google has invested in algorithms to fight click fraud. Ranging from simple to complex, these algorithms have been effective to a great extent. Some heuristics they may be using are:

• Look for burstiness of clicks on google ads from the same identity (cookie, IP address, Browser signature) across websites, or on the same site. Such users are tagged as fraudulent.
• Look for burstiness of clicks on google ads on a single site. This may be concerted effort to increase revenue.
• Infer other information like the referrer page, and look for lists of pages being created for explicit ad clicks
• Sense if ads are being clicked by humans or a program (by comparing click to non-click time periods

People (cookies) and sites committing click fraud are routinely warned and removed from the system. In effect, Google made good strides in detecting if clicks on ads are fraudulent and non-conformant with normal human browsing behavior.

Even if Google can detect fraudulent clicks with good accuracy, they are still hurting its bottom line. Every bad impression means Google’s servers were being used up for no return. Detecting click fraud (and not charging the advertiser) hurts Google’s profits. They detect fraud to ensure advertisers get good quality leads.

Bad unnatural traffic to a page is bad for all ad networks. Impressions are served with no follow up actions.

How traffic exchanges undermine the click-fraud detection systems

Traffic exchanges offer traffic in exchange for traffic. They do not have any incentive in stopping fraudulent traffic – since they mostly monetize only on their own intermediate pages. When a publisher installs a TE widget on his site, he senses that the traffic he’ll get back depends on what is outgoing. So he starts clicking on the article suggestions. Notice that every click is going across the web to very different sites.

Fraudulent clicks are dispersed across the web by a traffic exchange

Now if you are a receiving site – there’s a good chance that a good chunk of the traffic you are getting from the traffic exchange is coming from long tail publishers who are visiting your pages in the hope of getting something back. Since they are coming from all over the world, at different times – no click fraud system you have can detect them as fraudulent. They all appear as legitimate to you.

On the receiving end, fraudulent clicks are indistinguishable from normal clicks

Say, you are hosting AdSense on your site. A bulk of the traffic you are getting is from publishers who merely want to contribute a click, and are not interested in reading your content – or click on ads you host. This unnatural behavior ensures that the click through rates of ads shown on your site are very low. Many testimonials on the web have spoken of the bounce rate of traffic from exchanges being extremely high. This traffic often only increases your pageview count and does not contribute to anything else.

If a huge number of pages on the web are part of traffic exchanges, then Google suffers a bad CTR rate on most of its ads. Its ad targeting system penalizes ads that are not clicked upon. There’s a good chance that good ads get demoted because of impressions to bad traffic. Google’s bottom line gets diluted over time.

So, are traffic exchanges evil?

Traffic exchanges stand firm that they provide value to users, by adding diversity of content and ensuring publishers get unique users from new demographics. However, their inability to tackle fraudulent clicking by the long tail publishers makes the ad networks vulnerable to bad quality traffic. The bottom line of all ad networks – whether based on impressions or clicks is lowered by fraudulent traffic. This in turn lowers the monetization for participating publishers. A vicious cycle to death. Perhaps traffic exchanges can change their models to ensure high quality contextual traffic and disincentivize fraud. Perhaps ad networks should not show ads when traffic is referred by a traffic exchange.

We are happy to be receiving a very good response from publishers and bloggers to our content discovery service. At Dhiti, our vision is to provide hassle-free products that improve the reading experience on the content web. In this article, we’d like to address some frequently asked questions about our service.

1) Do you recommend only articles from the given hosted site in your widgets?

Yes. As of now, we only recommend articles from the publisher’s site. A publisher does have the control to extend this to a family of sites from his publisher network. Some sites like Hindustan Times have asked for this feature, and we are open to requests from other sites for this.

2) Do your widgets cause any load on the hosting servers?

Content discovery algorithms take a lot of CPU. This is why it has been restricted to big publishers who could afford the extra load. At Dhiti, due to our cloud-based solution, we do all the compute on our servers and ensure your servers dont take any extra load (but to serve the extra pageviews we can give you). The content we show in our widgets is directly pulled from our servers from the user’s browser. The reader sees no difference in experience – since our widgets absorb the same css settings from your page.

3) Do your widgets cause latency in page loading?

Latency is the biggest cause for user dissatisfaction. While we strive to make our service fast so as to not cause latency concerns, we go a step further. We load our widgets asynchronously, well after the rest of your page’s elements are loaded. This ensures that our widget is loaded after your page and your content is fully rendered by the browser. You may see our network connections being initiated a little later, but this causes no visual delay for your reader.

4) Do your widgets affect SEO?

Most of our widgets are loaded using script tags. Our content is rendered directly on the browser using Ajax. The related article recommendation widget does not contribute to SEO.

If you are also using our hubs feature, the hub pages will get indexed by search engines. When search engine bots visit the hubs, we send them the content – so they can index and make the pages discoverable on user searches.

5) Do you do user cloaking? (In other words, do you send different content for a search engine and for a user?)

Search engines dont understand javascript. To ensure content in your hub pages are properly indexed, when a bot visits the hub pages – we send them rendered pages. When a user visits the same hub page, the pages are rendered using Ajax on the browser. So both bots and your readers see the same content in the end. For end users, we use script tags and ajax for rendering. We do this to ensure low latency for your users, and no extra load on your servers.

If you are concerned about this, and want the same HTML to be presented to both search engines and readers, and dont mind the extra latency – you can disable the “Render using Ajax” option in your console.

6) Why do you use redirects for the recommended links?

Users are the ultimate deciders of relevance. If we present 3 articles to them, and they always pick the second one – its a clear indicator of their preference. We wish to use this in our ranking to ensure we present the most relevant articles at all contexts. To learn about user preferences, we send ourselves a signal – through the redirect.

The redirect also helps the publisher/ blogger know how well we are performing on his site. Look into your Google Analytics for traffic referred by http://dive.dhiti.com/redir.

7) Will Google ban my site because of using Dhiti?

Absolutely not. Dhiti does fair play. It is meant to improve user experience, and uses no illegal methods to boost traffic on the site. Dhiti does not steal SEO juice as well – and mostly renders links using Ajax. Dhiti uses best practices, believes in good SEO, and wants to make good content win. The techniques used by Dhiti for rendering are also industry standard.

Google News rocks. It has the ability to collate related stories together, giving orthogonal perspectives on the same story. How many people have wanted to do something like Google News for their own verticals? How many days would it take to do something like that?

Today, Channel Tech Center launched a curated portal for B2B industry news, ChannelToday.

Channel Today's B2B aggregated news

Take a look at that page. Its fully integrated into their site. Reflects their look and feel. Has their branding. It curates industry news on B2B. Every article is tagged with relevant concepts that drive the article, along with related news. Very much like Google News. How many days do you think this took for Joseff Betancourt?

2!

Yes, 2. Joseff approached us and said he loved our technology and what it can do. We piloted our content discovery widget on their site. Then powered rich hub pages on ChannelTechNews. He replaced Google Custom Search with ours. Our products were being put to good use. But he was still hungry. He wanted to have a curated portal of all B2B news from the channel network. He wanted to know if we can help him do this.

Busy in the past week, we just dropped him a link to our API documentation. We allow many ways to ingest relevant data (articles) into a document collection. Given a set of articles, we could bubble up the most relevant concepts. Given any articles we could mine other relevant articles, and the concepts that drive them. Maybe this can help get some distance? What ensued was some mashery, and some support from our side. Joseff wrote a cute PHP wrapper to our API, and used it to create his own Google News. His algorithm was pretty simple:

1. Pull up all the data he cares about into a “session”. A session is our way of calling a document collection.
2. Our text mining/ content discovery engine has already started working. Its pulled articles, indexed and understood them.
3. He just asks for the list of articles, sorted by time. He gets a JSON array for this that can be iterated through.
4. For every article in this list, he asks for related articles, and the top concepts that matter. Another JSON array follows.
5. Now its just about rendering the page based on all the information he got. To ensure good response times, he caches the page and hits our server once every 10 minutes for updates.

Pretty simple you’d say. Why did he take 2 days for this? Well, this includes the initial learning curve of our API, and its nuances – we’ll get better at this over time.

Is Joseff satisfied now? Not yet. He wants source ranking, the ability to train concepts, etc etc. He’s keeping us busy. But boy! Are n’t we happy?

A few days back, Dhiti launched the Help Dhiti Get Better campaign.  We  engaged with a lot of publishers to appreciate their priorites and to know what they thought about our products. We met some of our publishers personally. We spoke to some of them on phone. But what helped us reach out to the vast majority of our publishers was an online survey. Here are some of the key insights we gathered :

• 82.6 % of the publishers want us to provide  insights into what their readers are doing.

• 72.7 % of our respondents wish to write better articles.

• 72.2 % feel that we should support images and videos in our widget.

• 60.9 % want us to help them generate  revenue.

• 43.5 % have asked for an option to place Dive on the sidebar.

• 30.4 % would like to see a smaller version of the Dive widget.

• 26.1 % feel we can improve upon the existing layout.

We also got (handwritten )  feedback on various things, big and small.

Thanks for being brutally honest with us. We take your feedback seriously. In the days ahead, we will be launching a slew of exciting features and products, all aimed at taking content discovery on your site to the next level.  We shall keep you posted on these updates. In the meanwhile, if you like to read up on a few articles about engagement and reader behaviour, you are most welcome to visit us here :

Blog -  http://dhiti.com/category/blog/

Twitter – http://twitter.com/dhititech

Facebook – http://www.facebook.com/pages/Dhiti/115299665171226

If you wish to tell us something more, please write to info@dhiti.com

Everyday, you get to know things from many sources. You read news online, you are hooked onto Twitter, and RSS feeds. You visit your favorite site on the web. But no site can match the relevance of some one from social network says something to you. Whether its about a new music album, or a new movie. Whether its about a funny incident that happened, or about a political issue worth debating – you give the highest attention when you hear something from your social network. I am not referring to your Facebook news feed here. Your Facebook news feed is still about news being broadcast to the social network. I am referring to things someone tells you directly. Statistically, news, information and tidbits you receive directly from your social contacts are the most relevant to you.

Word of mouth has the highest relevance in information dissemination

Why is word of mouth so effective compared to other channels?

Social information filtering!

Social contacts constantly seek feedback, and learn what kind of news is appropriate. There is filtering happening at two ends.

Inbound filtering: When I hear something from Shradha (of YourStory.in) about publishing, I give it a higher weightage – than when she says something about search algorithms. When I hear something from my professor about a new type of auction, I am all ears – but not when he comments about the startup ecosystem. I apply a filter about what I give importance to. This is an inbound relevance filter.

Outbound filtering: I know what to say to whom. I will recommend an user engagement strategy to Shradha, but not about a new cool programming language. With my colleague, I’ll discuss the Bowling Pin strategy to acquiring customers. I know what they are interested to know about. I do outbound filtering.

Word of mouth communication is so effective, because of this dual filtering. Information is not broadcast to everyone. It is targeted. Similarly, everything thats received is not given the same importance. It is prioritized. The actors quickly learn and adapt. They do not spend energy shoving information down a channel thats not receptive.

So, whats the connection to the article’s title?

As a publisher, you are writing several articles. All of these are going down a twitter, RSS or Facebook stream to your readers. Your readers are already applying inbound filtering. They look at an article title, read a snippet and then decide if they should dive in.

Your readers: Do you know what they want?

But, are you doing your outbound filtering? Do you know your audience? Do you know what they want more? Do you know what they want less? Are you adapting to their taste?

Before trying to achieve the holy grail of emulating the effectiveness of word of mouth in the online publishing world, you must first get feedback about your audience’s taste. This article is about that!

Why should a publisher know what the reader cares for? To improve relevance. Every article you write takes some effort from you. Do you like these articles going down the twitter stream of your readers being overlooked? No. You want them to like and read as many articles. How can you adapt to their interests, drive up relevance and hence engagement on your site? By knowing their interests.

But how do you estimate their interest? Yes, I know you all have Google Analytics, and Google Analytics tells you which articles got more traffic. You also know the top queries that brought you traffic. With some effort you can pull out the articles that have increased commenting. But all if the information you have is about articles. Can you bubble up these patterns to the concepts that readers respond to?

I’ll organize this discussion around some questions for which you should find answers.

1) The content profile: Build a profile of the concepts you are writing about. These are the concepts you are investing on. They form the basis for which we’ll measure ROI and response.

Content is king

2) The traffic profile: Build a profile of your most popular articles. These are the articles that have kept Google Analytics’ servers busy. They are feeding your display ads too. Is the traffic profile the same as your content profile?

Traffic from different sources

2) a) Direct traffic profile: Build a profile of articles that are visited directly by your users. These users come to your website, and then read your articles. Which ones are they reading?

2) b) Social media traffic profile: Build a profile of articles that get shared on social networks. Sharing is the best response you can get from a reader. He’s not just happy, but is encouraging his network to read as well. Its important to separate the bots and trolls from the real users in the social networks. So instead of the actual act of sharing, we’ll measure the traffic your site gets as a result of sharing.

2) c) Search engine traffic profile: Build a profile of articles that get you traffic from a search engine. While social media can give you a measure of how interesting your articles are, search engines give you a measure of how useful your articles are. People search for their needs, and then click to come to your site. users from a search engine are purposeful, and state what they want clearly. They also present better monetization opportunities.

3) Exploration profile: The direct traffic you get is biased by your most recent articles, since you’ve placed them in prominent places on your website. The social media traffic you get is biased by how interesting the articles are. The search traffic you get is biased by the intent content match, and your competition. All of them tell some aspect of your readers, but seldom give the complete picture. However, when any of these users read an article, and explore your site further – you know what their interests are. Their first click may be influenced by your placement, their social content or immediate need. But their second click in your site helps see beyond their immediate context into their broader interests. What kind of articles are they exploring into?

The explorers

4) Comment profile: When a reader comments on your article, he’s increasing the wealth of your content. Comments are not just a great way of soliciting reactions from your readers, they also serve as ways to extend your content into a discussion. Not all of your articles get the same commenting. Some do. What concepts do people comment on? Are they indicators for your next articles?

Comments

A case study with YourStory

We did an experiment using YourStory’s data, and the information analytics engine behind Dhiti, and created these profiles. While they are only taken from a time limited sample of the site to ensure no competitive intelligence is revealed – we still had some great insights.

1) Content profile: What are the concepts YourStory is writing about in the last few days?

Entrepreneurship: Entrepreneurial ecosystem, Founder
Business terms: Funding, Customer, Corporate training, Leverage
Private equity: Venture capitalists, Venture investor, Business angel, Angel groups
Finance: Debt capital, Financial advisory, Public companies, Issuance, Accounting, Long-Term, Project financing
Academia: Intellectual, Academy, Undergraduate, Faculty, Alumnus, Term, Chair
Business: Business administration, Ensure, Exports, Business Development, Disappoint, Vendor, Business acumen, Nascent market
Industries: Professional, Distribution, Energy sector, Consumer Electronics, Governance
Marketing: Product planning, Software Development, White Paper, Advertising, High end, Product line, Distribution, Presentations
Motivation: Action plan, Incentives, Enable, Objective, Promotion, Energy, Hunger
Financial markets: Liquidity, Capital Market, External financing, Financial Services, Guidance, Valuation, Hybrid market, Long

2) Traffic profile: What concepts are people actually reading?

Entrepreneurship: Entrepreneurial ecosystem, Founder
Management: Resources, School of business, Professional, Entrepreneurialism, Management Consulting, Senior management, Head office, Business Management
Economics of service industries: Costs, Managed Services, SaaS, Service Provider, Suppliers, Quality, Computer Security, Certificate
Learning: Ability, Experiment, Studying, Learning management system, Learning Community, Lessons, Interesting, SKILL
Economic development, technological change, and growth: Productive, Emerging economies, Renewable Energy, Technological progress, Capacity building, Development, Slums, Rural infrastructure
Motivation: Innate, Incentives, Action plan, Enable, Object, Energy
Promotion and marketing communications: Self-promotion, Advertising media, Internationally, Poster, General public, Multi-media, Brand name, Direct to consumer
Maintenance: Maintain, Repair, Failure, Service,

2) a) Direct traffic profile: What concepts do people coming directly to the web site read?

Agronomy: Horticulturists, Farm Management, Agricultural research, Sowing
Bookstores: Bookstores, Book store
Business term: Customer, Corporate training, Funding, Leverage, WORK
Business models: Revenue model, Subscriber, SaaS, Service Provider, Sustainable business, TRUST
Branding: High end, Brand recognition, Brand name
Industries: Distribution, Professional, Energy sector, Governance
Accounting systems: Book keeping, Double entry accounting, Credit

2) b) Social media traffic profile: What concepts do people share and read from social media? To see the difference between Facebook and Twitter audience, the two are presented separately.

Facebook:

Promotion and marketing communications: Target audience, Advertiser, Coupons, Press Release, Reach
Indian masculine given names: Deepak, Harsh, Abhishek, Naveen, Raj, SHARAD, Kartik, Arun
Business models: Revenue model, Subscriber, Micro-Enterprise, Business Model, TRUST
Healthcare quality: Disease management, Health care services, Cost effective, Medical Care
Business term: Corporate training, Customers, Funds, Funding, Clientele, Leverage, WORK
Occult tarot: Tarot card reading, Tarot card reader
Motivation: Innate, Ambitious, Enabling, Energy, Resilience
Primary care: Primary Health Care

Twitter:

Entrepreneurship: Entrepreneurial ecosystem, Founder
Motivation: Actualize, Innate, Primary objective, Ambitious, Enabling, Morale, Inquisitiveness, Goal
Business: Businesses, In-house, Ensure, Selling, FMCG, Proposal, Firm, EXPORT
Business terms: Market Capitalization, Operating costs, In-house, Retail chain, Payment, Tycoons, Bespoke, Funds
Consulting: Management Consulting
Financial ratios: Capital ratio, Rate of Return, Leverage, Statutory Liquidity Ratio, Beta, Color
Financial markets: Financial market, Liquidity, Financial instruments, Guidance, Financial sector, Systemic risk, Private Equity, Financial products
Investment: Investment fund, Developed markets, Private Equity, Carry, Mutual Fund, Collar, Correction, Option
Banking: Investment Banking, Capital ratio, Banking system, Statutory Liquidity Ratio, Banking Business, Cheque, Free Banking, Bankers
Interest rates: Sub-prime mortgage

2) c) Search engine traffic profile: What concepts do users coming from a search engine tend to read?

Promotion and marketing communications: Promotion, Target audience, Advertise, Coupons, Sponsorship, Press Release, Buzz
Strategic management: Strategy formulation, Competitive Advantage, Management Consultant, Corporate Strategy, Leading, Leadership, Leverage
Types of organization: Non-Profit Organization, Professional association, NGOs, Institute, Socially, COMMUNITY, Foundation

3) Exploration profile: What concepts do users tend to explore on the site? Analyzing this was made very easy by the Dhiti widget on YourStory. Dhiti organizes relevant articles from the site into concepts making it easy for readers to dive deeper into them. When readers explore on a concept, they are giving away their interest to know more on the topic. The concepts readers dived into at YourStory are:

Entrepreneurship: Entrepreneurial ecosystem, Startup companies, Entrepreneurial education, Entrepreneurs, Business Plan, Creative entrepreneurship, Center for entrepreneurship and innovation, Small enterprise, Startup Weekend, Founder, Venture Capital
Business terms: Business models, Business acumen, Business idea, Business Development, Technology Business Incubator, Customer base, Privately held company, Arbitration, Strategy Consulting, Sustainable business
E-Commerce: Group buying
Marketing: Marketing strategies, Online marketplace, Display advertising,
Software Development: Software Testing
Rural economics: Sustainable business, Social impact, Wave power

4) Comment profile: What concepts in articles seem to invite more commenting from user?

Promotion and marketing communications: Target audience, Advertising, Ratings, Sponsor
Business term: Customers, Funds, Funding, WORK
Marketing: Marketers, Target audience, Advertising, Discounts, Pricing, Consumers, Win-win situation, Service
Renting: Rentals. Car rental companies
Advertising: Advertising, AdAge, Sponsor, Attention
Blogging: Micro-blogging

Analysis

With a reasonable amount of data, its time to put our analysis hats on, and draw inferences. The intent here is not to judge the audience of the site, or the content on YourStory. The sample taken is intentionally small – so as to not reveal broad patterns. That analysis should be confidential to the owners of the site. This inference exercise is mainly to observe differences in information consumption based on the channel of discovery. They would reveal different patterns for different sites giving clues towards content strategy. Notice the following patterns here:

• YourStory’s content production is clearly skewed towards Entrepreneurship, and Venture capital. However the direct traffic from their home page tends to prefer outlier areas. This may be indicative of the type of stories they “feature” on the site.
• Traffic from twitter is mostly interested in technical aspects from the site.
• Personal and social aspects (like health care, entertainment (tarot reading) tend to be popular in Facebook.
• Search engine traffic shows the presence of social entrepreneurship – for which YourStory has good search engine presence.
• The exploration profile matches entrepreneurship and venture capital the most – which is the focus area of the site.
• Topics that tend to have more public opinion get more commenting (rentals).

Applying outbound filtering:

Getting back to the main theme with which we started the article – outbound filtering. The different channels clearly show different patterns of consumption. YourStory’s direct audience wants features – stories about outliers. The Facebook audience is interested in personal and social stories. Twitter audience seems more tech savvy. YourStory’s search engine presence is skewed towards a different area. How can YourStory make use of this to improve relevance across all channels?

1. Concentrate on “features” on the web site’s home page. Get more stories on outliers.
2. The twitter audience is more interested in technical articles. Feed them more of that. Perhaps curate your audience by sharing relevant content from elsewhere in the web.
3. The facebook audience is more personal in nature. Perhaps thats the place to “connect” to entrepreneurs.
4. Depending on whether YourStory is happy with its search engine profile, it can pump up more content in that area, or invest in SEO for getting better coverage for its other content.

So after so much analysis to know about reader intent, we must not forget the most well-tested method of all. Just ask! Have you considered doing surveys with your readers?

“I need to do SEO”, is something you hear often from an aspiring publisher who has produced a lot of content, but is not seeing traffic grow.

“We can increase your search traffic by 400%”, is something you hear often from “Search Engine Marketing Consultants”, who prey upon the ignorance of the aspiring publisher still groping in the Internet forest.

“I hear they have a team of just 4 people and get a million hits a month”, says an aspiring publisher from an old-school of thought who is completely aghast that some under-graduates are able to write content that draws huge traffic.

If you’ve heard these, or can associate with them in some way, and want to understand the underpinnings of this phenomenon, then this article is for you.

Old-school publishing was mainly about the actors, not about the consumers.

To understand what this means, pick any news paper. What do they write about? They write about events, about people, about trends, about brands, products and companies. Journalists mostly hang around with the news makers. They dig for the news, and bring the news to the rest of us.

Mainstream publishing is about the actors and news makers

Now lets go to the other end, and look at what eHow does. They publish a lot of content too, but they publish content – not about the actors, not about the news makers, but about you, and your needs!

eHow publishes news about you, and your needs

A site on How tos? This would not raise an eyebrow. How much traffic can a How to site make? Here’s the spoiler – at the time of writing this article, Demand Media, the company beind eHow has a market cap of 1.86 Billion, and NY Times has a cap of 1.56 Billion.

What exactly happened?

The publishing industry has always preferred to accompany the news makers – with elite editors deciding what common people need to be reading. But a common housewife needs to know tips on organizing her closet more often than knowing about an earthquake in New Zealand. Smart content producers like Demand Media have understood common needs better, and produced content more people need most of the time. The strategy worked over time. Demand media is the flag bearer of an intent-targeted content industry that is taking the World Wide Web through the next phenomenon.

Intent targeting trumps branding

We always got our news. Maybe we did not get it the minute it happened, like on Twitter, or the other up-to-the-minute websites. But news always got to us through dailies, or weeklies. However, we never got timely information for our needs. “Do I need a separate visa to visit Switzerland?” “How do I unlock the iPhone?” “How do I give bath to my cat?” All this changed with the world wide web, and search engines. First, with blogging the cost of publishing went to zilch. Next, with search engines the cost of searching went to zilch. Since content was easily discoverable, more common people started writing. Our lives changed. We started to consult search engines for all of our needs. “10 ways to sleep?” – yes, there are pages for that. At any point in time, our hunger for information for our needs overshadows our need to know “the news”.

A query like "iphone unlocking" does not have any brand presence

Even before Demand Media got into the limelight as a content farm, there were thousands of cottage publishers – 3-10 man armies that were producing content rapidly. Try a query like “unlock iphone” on Google. Do you see many big name sunbathing the results? Imagine the millions of queries like this that are left untargeted by the branded publishers.

To get traffic to his site, an aspiring publisher must concentrate much more on what people want to know than the news makers. People will come to your pages chasing their needs. You can cross sell your other content from there on.

On the web, targeting the intent of users has a better ROI to draw traffic than investing in branding. While we are at it, maybe we should understand how branding works for online publishers.

Online branding for publishers

If you are not one of those few who went through all the results in the last image, please read it again. Do you recognize any publisher there? Yes, you do. PCWorld. Now thats a brand. Branding for a publisher directly correlates to the name recall or recognition. Does the name of the publisher instill confidence and trust about the content in the article? In behavioral studies we have often found that people often scan for the publisher name, and click on the article from a better known brand. This would perhaps give PCWorld an advantage even though there are other high ranking articles from less known sources. What does it take for PCWorld to become a brand online?

• Publish high quality content that targets user intent. For a page to show up on a search engine, it must actually address topics that people query for.
• Ensure the user experience on the site is top notch. A bad user experience makes users hit the dreaded back button faster.
• Invest in good content discovery tools on the site. For every second article a new user reads, brand perception quadruples. [Shameless plug: Do you know about Dhiti?]
• Ensure that many other sites link to this site, and improve PageRank. Having higher PageRank increases the probability that an article is shown for a relevant query.

Good indicators for publisher brand image are:

• Direct traffic to the website. How many people type pcworld.com directly on the browser and come to the site?
• Navigational queries that lead to the website. How many people type pcworld in Google and then hit the first result they get?
• RSS feed subscribers. How many people have subscribed to the RSS feed to get updates on their site?
• Twitter and Facebook subscribers. How many people subscribed to the site’s twitter and facebook profiles?

All of these are indicators that people care about the site even if they dont have a specific intent.

Traffic = Intent content match + PageRank – Competition

Lets delve a bit into how search engines work. Search engines typically come up with a notion of the brand of the publisher – PageRank is used by Google, other close variants by Bing and Yahoo. However, having a high PageRank does not ensure the site gets all the traffic in the world. The content on the page must match the query. This is a pretty strict condition. Search engines like Google give an extremely high importance to the query match. For most queries, PageRank just ensures that a given page comes into reckoning to compete among other pages. Once these pages are in the top pool, the goodness of the match between the query and the content decides the ranking. Why do you think the page from PCWorld was ranked lower than the others for the query “unlock iphone”? PCWorld surely had much greater PageRank than iphone.unlock.no. Yet.

The importance given to the query-document match leaves a window of opportunity for small players to write content targeted better at queries, and draw a lot of traffic from Google.

So, what are your chances of getting traffic for a query like “unlock iphone” if you are starting off today? You are low on PageRank. The space is also overcrowded with zillions of articles on the topic. Pretty much every combination is taken. So, you must choose those areas of intent that have less competition.

Lets discuss this with a pictogram:

A pictogram explaining why you get (or dont get) traffic from a search engine

People’s intent to know is always bigger than what is ever written. So, the circle representing user intent is huge. Your competition has a good part of that intent already covered. The part you get search traffic for is the intersection of your content with user intent, that is not overlapping with your competition. Do you normally see a big chunk of your articles that are never blessed by a search engine? Thats because it was either clouded by your competition, or people never really wanted to know about it. Lets say, you wrote a fantastic article about a cat crossing the road. Pity, nobody wants to search for it. Its doomed to your shelves. Or did you write about apple launching another great product? Thats great, but a thousand other people already did the same thing. Perhaps you have a chance if you write about jailbreaking the iphone using redsn0w. Do you get it? You have to write about stuff that people look for and not yet covered by many others.

Queries are getting longer

Google has caused behavioral changes in way people seek information. First, by building such a fast search engine, they made it very cheap for trying different query combinations. Next, they have not really invested in faceted navigation in a big way to help people digest hundreds of result pages. They further nailed this when they released Google Instant. The cost of people trying out query alternatives is extremely low. Google provides query suggestions and shows results as you type. In such a fast paced user interface, do you think people are looking beyond the first few results? Google has made people try out more queries. This is again an opportunity to aspiring publishers. When users try a query like “iphone unlocking”, they are not going to the second page to look for an article that talks about using iphone 3gs unlocking. They inspect the first result, figure out its not relevant, and refine their query to include 3gs. The big move by Google to solicit more intent from users has further reduced the importance of PageRank and increased the importance of casting a wider net to capture user intent.

Fighting the battle for traffic

I used the word right. Its not a war, but a battle. To get search traffic, you do not wage war with an entire site, but small battles with other peers in search result pages, for those queries.

To get ready for the battle, we must first assess the ground situation. Lets profile the stakeholders first – your targeted users (and their intent), the content you have (or plan to write), the competition (what has your competition already written about), your reader interest (what do your readers explore on your site).

Having all these profiles handy will help your content strategy. For instance, if you know your content profile, you can figure out what users query for around that. If you know your user intent profile, you can figure out who your competition is.

Content strategy: Create profiles of your stakeholders

Lets say you are a site focused on real estate in Bangalore. What would you do?

1. Gather all the topics that you wish to write content about. This seeds your content profile. If you already have written a lot of content, find the broad topics you have written about. You can use a concept discovery engine like Dhiti for doing this.
2. Observe what your targeted users are querying for. Use some seed queries from your content profile, and look for variations of these that are popular in search engines. Google’s query suggest already reveals a lot of information on this. A seed set can be easily grown to 3-4 times the size.
3. From these queries, learn about your competition. Your competing pages are simply the results that are already up there in the search engine. Observe what they write about – bubble up the concepts, and grow your competition profile.
4. If you already have a site which readers are visiting, then your readers are leaving trails of interest already. Have you ever checked what people search for inside your site? Have you checked the kind of topics that gather more comments?

With a good assessment of your ground situation, you are now equipped to go and write high-quality content.

For people with more financial muscle, there’s another way out to gather user intent. Buy user traffic from ISPs. Many ISPs are greedy and willing to sell anonymized user traffic for a price. This is worth more than gold. You know everything about what people are looking for. You can write content tailored to their queries. This is pretty much what Demand Media does.

Addressing the topic of gathering user intent, and the various available tools for this deserves another post. Listen on!

The next time an SEO consultant tries to sell this as rocket science, you know what to do.

A search user landing on a particular page is a fairly special event. How does Google treat such a user compared to publishers? Google is a transactional system, where the motive is to find something quick, whereas a publisher site is an engaging and influencing platform. Even though the motives are different, there is an overlap in learning, technology and means of monetization.

The Google side of the story

Google, is the Jeeves for all us Woosters. Practically, the whole world goes to google for help and Google has tuned its system to serve this use case very well. Not just in terms of technology, but also in terms of people’s mindset. If you dont find what you are looking for, “This is all your fault, Wooster”, then the user is supposed to refine the query. Even Steve Jobs could not pull that off with antennagate. Google does help the user in quick query formation and refining it if necessary (google instant). It is in Google’s interest, to help the user, type a query as close as possible to his intent. If the query is better servered with ads, google will try to nail him right there. See the example below for a query on “laptops”

If the user ends up clicking on an article link, it is fair to assume that he is seeking information. This is where Google lets go (or so it seems) and expects the article page to serve the user. Google is an transactional system and is very good at it.

The Publisher side of the story

For a given search query, the page and the site should have ranked high for that query.  Even after the search engine ranks it high, the user should decide to click on the link. If a huge probability math works in the publisher’s favor, the user lands on their article page.

A search user could be in one of the following states,

1. When he is still unsure if the page satisfies his intent. This is the stage when he reads the title and maybe the first passage and has to decide whether to read further or not
2. When he decides to read and finishes reading the article

Before we go deeper into understanding more about these scenarios, I want to bring Google back into the picture. Having lost the user to a publisher site, google will not let go of the user. They still play a very important role. Google spends insane number of compute cycles and man hours to come up with adsense ads, which have such high percentage of click throughs that there is no competition. It is in Google’s best interest to lure the publisher to highlight ads and publishers are more than glad to do so.

To make it worse, many publishers display  adsense widget just after the title, even before the content of the article. I am sure an ad serves best for certain kind of users, but not in all cases. It is in Google’s interest to always show an ad and lure the readers towards the ad, but is it in the publisher’s best interest?
Before the user has read the article, he is still in the evaluating mode. Now is the time for the publisher to impress the user, rather than distract him away to an ad. This is the time to influence and engage him more. The control on what should get more attention (ad or content) should be decided by the intelligence that the publisher builds about the users’ reading habits and not Google. An user’s intent can be varied, but it is for the publisher to understand the motive (as much as possible), and try to best serve the user’s intent. The user’s intent could be

• Information : eg., “best tech companies to invest in 2011″ –  serve him well, serve him fast
• Exploration: eg., “thesis driven investing” – introduce the concept and help him walk through the concept
• Learn in depth about a certain subject: eg., “help me get more traffic to my blog”:, where a digest around a related material should be displayed to enable learning in depth

Notice how Google knows it is better to show images for this query ahead of other results. Similarly, where is the learning on the publisher side?.

Here is an example of someone who has learnt it well. This link is the first link on Google search results page for the query “site:amazon.com running shoes”. But “running shoes” is too generic a keyword, so Amazon has decided that it should showcase more of its links.

Understanding intent and building intelligence is as important as any other part of the content strategy. People will continue to articulate their intent in a set of keywords, and its time for publishers to start taking keywords seriously (not just for seo). Understanding people’s expression, their intent and their behavior will help publishers become effective in serving the user and also learning to monetize closer to how they influence their users.

A search user landing on an article and also reading it completely is the best thing to happen to a publisher. Once he has read the article he is in the reflect/iterate/stop phase, so unless he wants to stop, you can continue to capitalise on this user. To do so,

• Suggest good reference links to read
• Request the user to endorse your article by sharing it
• Engage him in a discussion around the article
• Invite the user to sign up to your feeds/twitter

The user may want to continue reading around the same subject or he would want to explore your site.  Dont link bait him to read any random article. Engage with him to serve his current intent, and he is then likely to come back to you. This demands understanding intent, and serving the intent well. In a followup post I will try to write about tools which the publisher could use to address some of these concerns, like

1. daylife, for topic pages
2. juiceanalytics, checkout their keyword trees visualization
3. disqus, commenting
4. dhiti, content discovery, content strategy and topic pages(in beta)

It is the ideas. The ideas that you write about. As a reader, every time I am introduced to a new thought, I have this feeling of joy. When you have read a fantastic book, or a beautiful poem or an insightful blog, and, you sit back to soak it in, these are the moments which bring a lot of joy. The expansion of your knowledge brings in a queer sense of achievement, it feels like your soul has grown. The users to whom you incept an idea is your best friend. He is going to share your links, talk about you, subscribe to your feed and hold your site in high regard. In short you have an influence over this user.

An idea/thought is precursor to action, while business needs have made publishers focus on what the user does, while what the user feels has been neglected to a very large extent. But there should be equal focus to help users in what they are looking for

As content providers, do you do enough, to engage and influence your users. Do you

1. Surface your ideas to your users
2. Surface the interconnections between ideas
3. Let your users interact with your ideas
4. Monetize in sync with most important asset of yours (shall be taken up in a different post)

The interesting part about ideas is that, what each person takes from the same content is different. Ideas are about interaction of concepts with an individual’s knowledge. They  spark a different reaction in each of us because we all arrive at similar content with different states. Matt Ridley puts in across very eloquently in this talk.

So when an apple drops off a tree, there has to be a person sitting there thinking about how the celestial bodies orbit to come up with Theory of Gravitation. So its unlikely that you will influence all your users, but you can influence many of them, and for that you need to provide them with the right tools. Gathering an idea from a site is fairly similar to explorers of medieval ages. They dont know what they are going to get to, but they need all the right tools, to get wherever they need to get to. Your users need similar tools

1. A North Star, to keep tab of where they are
2. A map of world, to know what is where and what else is out there
3. A viewing device, so see from a distance
4. A ship, to navigate.

Translate it to a content site dynamics

1. The context of the user is important, the search query that brought the user in, his known interests, the article he is about to read or has just read. The user is likely to be influenced around topics in his context
2. What are the other concepts that the publisher has covered. And again these have to be relevant to the context the user is in. It needs to be surfaced at the right depth and breath. And user should be able to choose to hop across concepts or within a concept.
3. A quick way for your user to make decision, exploration is a function of effectiveness and time. If we show a nugget, most important around the idea, so that the user could quickly decide if he wants to read further or not.
4. Effective navigation, user should be able to interact and surf between topics, concepts, slicing and dicing them. Jumping and hoping, because you never know what interests them.

A user, for instance Charles Darwin takes from Population, Famine and Disease is very different from others. This theory inspired him, in coming up with his “Theory of Evolution”. The idea that changed everything. Connecting manually is innovation. But facilitating these connections is engagement.

In today’s world, where there is an explosion of information and its related concepts. It is difficult for an individual to keep tab of all connections. He needs assistance in suggesting relevant connections.

Here is a article about the surface of the moon and while reading this, it would be delight to connect this to ancient indian beliefs about eclipse and connect it back to celestial bodies in orbits. These connecting of dots is what makes it a joyful experience. Let you users get this feeling of epiphany.

Surfacing your ideas around a users context, letting them navigate effortlessly, letting them interact with your ideas and connecting the dots will help you be the platform for ideas.

These thoughts have inspired us and we tried to put it together in the following video

“I didn’t come here to tell you how this is going to end. I came here to tell you how it’s going to begin”

There are times when users on a content site have an urge to explore. They are perhaps excited by the article that they just read, and they are raring to explore , either on topics related to the article or the whole site. But the navigational aides are geared towards a sober approach of finding an article, not a quick way to explore.

The usual navigational aides provide directions at best, but not a gps, compass and a map that an explorer needs. We at Dhiti, are excited by the problem of how a user can quickly grasp what a site talks about and get a kaleidoscopic view of the entire site.

We drive this in two steps.

1. We generate context related to the current article the user is reading

2. Every context we generate can be pivoted on, to help the user understand its surroundings.

In the image below, the article is about angel funding in India. We have extracted a series of related articles, but we also show the concepts and topics related to this article. Now a user could pivot on either an article, a topic or a concept and our widget will always show context around his pivot

is the enlarge button on the widget to popup a bigger size window for better experience.

This opens a powerful tool to help him get a fantastic navigational experience to explore  the breadth and depth of the site.

We have found that many publishers themselves are excited by this experience, they get to see their own content in new light.  This experience is now available to all the readers as well.

Does it remind you of “Lady of Shallot” and her magical window?