Originally created 02/21/01

Many Web hits attributed to 'bots'



This might come as a shock to people who saw those Super Bowl commercials with the rampaging squirrels, but advertisers hate to throw money away.

Like other business people, they expect a return on investment. They particularly like to know ahead of time what they're buying.

That isn't happening on the Internet. Advertisers who measure everything in "cost per thousand" viewers or listeners are struggling to calculate the impact of their Web budgets. That struggle becomes more pronounced in the face of increased application of computer "robots" and "spiders."

Estimates of computer robot effects on hit rates vary widely. One Denver advertising executive said it might skew the rates by less than 1 percent. Another exec offered 50 percent. Still another went as high as 80 percent.

In other words, eight of every 10 "hits" to a Web site would come from a computer, unleashed by a search engine or a business gathering information.

Those computer robots, sometimes called bots in the industry, don't buy Pampers for their baby bots, and they don't drive Chevy Blazers to their virtual vacation homes in the cyberspace mountains. But Procter & Gamble and General Motors receive reports indicating that these bots visited Web sites and viewed their advertisements, as if the bots were actual consumers.

"It's something that we have done a lot of work on," said Michael Christian, senior vice president of WebSideStory, an Internet measurement company in San Diego. "It's not only an issue, but an issue that has been growing over time. There are two reasons. First, the volume of robotic traffic has gone up, depending on the site. Second, advertisers and business people are becoming aware of the fact that a lot of their statistics are skewed."

Robots often leave behind clues of their activities. Web measurement companies notice surges of activity in their server logs, with systematic "hit" trends through each page of the Web site. Some Web measurement companies, such as MuchoInfo in Denver, devise programs that require human interaction. MuchoInfo provides a random, pop-up survey on Web sites that robots invariably ignore.

But most measurement systems aren't sophisticated enough to detect robots with any degree of certainty. And some programmers build robots to avoid detection.

"It's hard to make discernible differences," said J.P. LaFors, direct marketing strategist for Thomas & Perkins advertising agency in Denver. "It's very easy for a robot to be invisible."

One indication of the increased presence of robots is the reduction in "click-through" rates on banner ads, said Mitch Bennett, chief executive officer of MuchoInfo. Whereas computer users three years ago tapped into those screen-wide advertisements at the top of Web sites at a rate of 3 percent to 4 percent, today that rate has slipped to about one-half of 1 percent. Computer users have become more savvy about most efficient surfing methods, realizing that the banner ads only slow them down.

"In the beginning, people were amazed you could click through," said Dan Barnhart, manager of interactive at Barnhart CMI advertising in Denver. "Now you don't even look at the top of the page."

But robots and spiders also affect the click-through percentage, driving up the visitation rate while avoiding the banner ads. The bot gets to the site, and the spider goes to work, burrowing as deeply into the site as necessary. Some tech experts refer to this activity as "spidering."

Despite these issues, advertisers' interest in the Internet has increased, said David Henry, president of Henry Gill Advertising. "What we now have," he said, "are clients who two years ago weren't involved in a very large way in the Internet, (but) who want a presence on the Internet. Unfortunately, the cycle has already passed them. In most cases, we've seen studies that banner advertising has decreased substantially in its effectiveness. And not only are the hits down, but you have to ask how many are humans actually looking for information?"

Essentially, the Internet has had no effect on the advertising industry's basic approach. You do as much research as you can, and when it comes time to make a decision, you go with your gut.

"It may not be the most perfect advertising medium of the future, but that doesn't mean we can ignore it," Henry said. "We have to figure out a way to be ahead of the trends... ."

Barnhart said advertisers can actually turn the tables on robots, anticipating their activities and enticing them into listing the advertiser's site at the top of their indexes. "The more you can trick bots, the more free advertising you can get."

In other words, if you can't beat computer robots, try turn them to your advantage.