Jul 10, 2012

Lies, damned lies and CSA data baiting




The Common Sense Advisory (CSA) has announced another rates survey for translation providers - would anyone like to hazard a prediction of the results that will be reported?

As a child, I was subjected to the annual school ritual of standardized testing, as most of those who pass through the US school system are. Around the third or fourth grade I noticed something rather odd about the tests. Aside from a puzzling tendency to repeat the same question material from year to year,the answers were often obvious even for questions I didn't understand. My scores on these standardized tests were more a reflection of my memory and ability to perceive bias than they were of any understanding of the subjects tested. The makers of the test had no particular stake in my choice of any given answer; it's just devilishly hard to avoid pushing someone toward what you perceive as the "right answer". Any good teacher knows how hard this is. That's also why we have double-blind testing and other methods in our often unsuccessful attempts to avoid experimental bias.

If honest people have a hard time constructing a valid test or survey, how much harder is it for those who have a particular agenda to do so? Do they even really try? The CSA bills itself as an independent Massachusetts-based market research company ... [which helps] companies profitably grow their international businesses and gain access to new markets and new customers ... [with a] focus ... on assisting ... clients to operationalize, benchmark, optimize, and innovate industry best practices in translation, localization, interpreting, globalization, and internationalization. Wow. Quite an orgy of jargon to describe what many perceive as a predetermined agenda of special interest propaganda. Like the sellers of another Watchtower, in recent years, the CSA appears to offer salvation.

As I'm sure they do. But for whom? I see the CSA quoted most often by industry shills for machine translation, those who follow the old formula of repeating lies often enough that they are eventually taken as gospel. Rates are dropping, rates are dropping say some of the self-selecting four percent who have responded to past CSA surveys. The fact that my own observations confirm at least the German part of the CSA statement that language pairs involving French and German were the only ones to escape substantial price drops is not prima facie evidence that anything else happens to be true or even that my own observation is generally correct for all market subsegments. In fact, some I know would argue that German translation prices have indeed been depressed in recent years. Statistics drawn from a tiny opt-in minority in a survey are simply incapable of offering any real insight into the truth of either undifferentiated statement, and one would have to be foolish or dishonest to maintain otherwise.

Now the CSA has sent out the call for the four percent and perhaps a few more suckers to share their financials, with the support of agency associations such as the Dutch VViN (formerly ATA, the bunch that try to convince translation agencies that they must get on the MT boat or drown with propaganda fests like the conference in Ede in the Netherlands) and others. The current survey is an "honor system" attempt (i.e. unverified, unaudited) to draw out the details of price policies from participants (freelancers and translation agencies are among those targeted - the CSA rejects the definition of an LSP in the EN 15038 standard and applies it only to companies with a certain number of employees). But there is no mechanism to prevent some monkeywrencher from filling the survey out any which way and skewing the results to paint a picture bearing no resemblance to reality. And even if there were, the tiny participants sample is hardly representative of industry trends as a whole. If I were to look for more reliable sources of data, I might start with some of the mandatory reporting data in certain national jurisdictions, such as that gathered in Germany by the Statistisches Bundesamt. Or try something approximating a recognized method of random sampling or introduce some kind of data auditing. If I felt that these data had any more value than to promote a set, self-interested charlatan agenda of machine translation, I might make an effort to gather credible data.

Rate surveys without a particular ideological or commercial agenda from the ITI and IoL in the United Kingdom and the German BDÜ have larger samples of service providers in many languages, and they mostly tell a story of stable or slightly increasing rates for language service providers.

But if I were paid to get results for my co-conspirators....



3 comments:

  1. I understand your concern and frustration. When I organize the first pricing survey, I made sure I had 12 buyers as a control group. These where companies that agreed to share how much they paid for translations. I don't know what is their current methodology.

    ReplyDelete
  2. I've just seen this rates survey. It is remarkable how ill-informed (or plain stupid) some of the questions are. For freelancers, for example, there is a question about "what is the fair price for a CAT tool" (with radio bottoms going from "$100 or less" through "$ 901 or more"): A badly designed underfeatured CAT tool could be overpriced at $ 50, and a reliable tool with a rich feature set, good technical support and frequent updates could be a bargain at $ 1000 or more.
    Or the question about the rush rate charge: you can only choose one specific percentage (whereas, for example, I apply different surcharges depending on such factors as the deadline, how much there is to do, and so on).

    ReplyDelete
    Replies
    1. I missed that one, Riccardo, but then I decided to be a huge agency the first few times I took the survey just so I could explore and document some of that stupidity. It is amazing how amateurish the design of the survey is. A lot more thought went into the ITI/IoL and BDÜ surveys, and they sell their reports for a lot less than the CSA!

      Delete

Notice to spammers: your locations are being traced and fed to the recreational target list for my new line of chemical weapon drones :-)