memoQ: stopwords for term extraction

The recent Kilgray blog post about the "terminology as a service" (TaaS) project reminded me of the considerable unfinished business with the term extraction extraction module introduced three years ago with memoQ 5.0. It's a very useful feature that I apply frequently to my projects, and my prediction years ago that it would not replace SDL's MultiTerm Extract in my workflows was wrong. Overall it proved to be more convenient, and after the shock of discovering that the defective logic of MultiTerm Extract created new German "words" that neither existed nor were in my text sources, I dumped that dodgy tool and stuck to memoQ's extractor. But sometimes its rough edges are irritating, and I wish Kilgray would finally pick up the ball that was dropped after a great start in the game.

One of the major weaknesses (aside from never remembering my changes to the options or my preferred settings for extractions) is the management of stopword lists.

Overall, Kilgray's approach to stopwords is reasonably sophisticated; some time ago I published a rather incomprehensible post in which I demonstrated how the "stopword codes" - those three digit binary numbers which appear stopwords - control whether a word can appear at the start, the end or the middle of a phrase even if it is excluded as a single word. These codes are quite useful in some cases. However, they also complicate the use of stopword data for most users.

memoQ includes only a few stopword lists for a few languages in its shipping configuration - German, English, French and Italian I think. Not even all the user interface languages are included. That's rather sad, because there are quite a few public domain stopword lists available on the Internet. However, most memoQ users have absolutely no idea how to incorporate these in memoQ, and Kilgray offers no features or information that I am aware of to facilitate this process.

I was reminded of this problem when I was asked to discuss terminology mining with masters students at my local university in Portugal. I thought it might be nice for the students to be able to make use of the stopword lists they could find on the Internet for their target languages (Spanish and Portuguese for that group). These lists are typically just text files of single words. When I build my own master stopword list for German a few years ago, I gathered half a dozen or more large, mostly redundant lists for a start. Then I carried out the following steps:

1. Combine all the stopword lists for a language from various sources into one big text file.
2. Open that text file in a spreadsheet program such as Microsoft Excel.
3. Sort the list and use the integrated function to eliminate duplicates.
4. Fill the number "111" in the second column of the spreadsheet. (This will completely exclude the term from phrases as well; if you want to make individual exceptions according to the scheme I described in an earlier blog post, you can do so now or at any time later after the list has been imported to memoQ.)
5. Save the data as tab-delimited Unicode text.
6. Open the text file and paste in this XML header, adapting the red parts to your particular list:
   
   <MemoQResource ResourceType="Stopwords" Version="1.0">
     <Resource>
       <Guid>dc7006ad-8db8-4724-b22d-7acfd600fd9f</Guid>
       <FileName>ger#KSL_stopwords-DE.mqres</FileName>
       <Name>KSL_stopwords-DE</Name>
       <Description>Combined lists from many sources</Description>
       <Language>ger</Language>
     </Resource>
   </MemoQResource>
   
   
7. Save the file, change the file extension to MQRES, and import the file as a stopword list in the memoQ Resource Console.

Afterward, the stopword list you imported will be available  when you start a term mining session using Operations > Extract Terms...

During an extraction session, words can be added to the chosen stopword list (one at a time unfortunately - I've been asking for multiple addition as a productivity measure for 3 years now so far) by selecting a term and Clicking the Add as stopword command or pressing Ctrl+W.

You might be a little confused if you look for words you've added to a stopword list from the term extraction interface. They are not inserted in alphabetical order, but instead at the end of the words starting with a given letter. Thus, for example, the red box in the screen clipping below shows all the words I've added to my previously alphabetized list since I created it:

memoQ&A: How can I have multiple target languages in a project?

Every translation environment tool has its own approach to project management and languages. Recently, a colleague asked for advice on outsourcing options for a client project which may involve a number of target languages translated from an English source text. I think her usual tool is some flavor of SDL Trados Freelance, but the five-language restriction on freelance software versions from SDL means that Studio is not an option in this case without an expensive upgrade to SDL Trados Studio Professional. This might be a worthwhile option, though it's difficult to judge in this case, because I don't know if this sort of project will be part of the future routine.

As I recall from my days as a Déjà Vu user, projects could have multiple target languages, but without the Workgroup version, collaboration was made difficult by restrictions on the bilingual files ("external views"). I don't know if this is still an issue in DVX3, because the uncertain future of Atril makes me reluctant to invest much study time in that tool.

memoQ does not restrict the languages one uses, but creating a project with multiple target languages requires the Project Manager edition. Kilgray doesn't talk much about this edition of memoQ and usually tries to link it to a server license sale (or a memoQ cloud subscription); if you look on the Kilgray web site, you will find very little information. I think the license typically costs about €1500, with €300 (20%) annually for maintenance upgrades and support. Quite a bargain, actually, and with the PM version one also gets access to another well-kept secret: the memoQ client API, which can be used to automate or simplify a number of project management processes. And of course with the PM edition one can create convenient "handoff packages", which provide convenient resource bundles for collaboration with other memoQ users with compatible versions of the translation environment tool.

But the memoQ PM edition is total overkill for someone who typically needs only individual language pairs for work and is faced only rarely with the need to manage multiple target languages from a source language. If there are just a few languages involved, it's probably not a big deal for someone with the memoQ Translator Pro edition to create several projects (one for each target language needed). There is, however, another option worth considering in some cases: using the Translator Pro edition to manage a multilingual project created with the PM edition.

If a multilingual project is created with the memoQ Project Manager edition, the individual target languages are available in the memoQ Translator Pro edition. A text imported with the target set to "All languages" is automatically put in every target language in the project; suitable bilinguals (XLIFF, RTF bilingual tables or TWB-/Wordfast-compatible unclean DOC) can be generated from each individual target language.

A downloadable example of a memoQ project backup to test for memoQ 2013R2 is here.

Using a project with multiple target languages in the Translator Pro edition requires, of course, that the original project be created by someone with the Project Manager edition. And the sophisticated functions for reporting, package creation, etc. are not available when the project is open in the Translator Pro edition. But translation memories, termbases, LiveDocs and other resources can in fact be managed for multiple target languages in the project. The work can be reviewed or prepared for external proofreading, target files can be created in all the languages, and a number of other simple management tasks can be performed. In many cases this may be enough.

The curious case of crappy XML in memoQ

Recently one of my collaboration partners sent me a distressed e-mail asking about a rather odd XML file he received. This one proved to be a little different than the ordinary filter adaptation challenge.

The problem, as it was explained to me, seemed to involved dealing with the trashed special characters in the German source text:

"Special" characters in German - äöüß - were all rendered as entities, which makes them difficult to read and screws up terminology and translation memory matches among other things. Entities are simply coded entries, which in this case all begin with an ampersand (&) and end with a semicolon. They are used to represent characters that may not be part of a particular text encoding system.

At first I thought the problem was simply a matter of adjusting the settings of the XML filter. So I selected Import with options... in my memoQ project and had a look at what my possibilities were. The fact that the filter settings dialog had an Entities tab seemed like a good start.

This proved to be a complete dead end. None of the various options I tried in the dialog cleared up the imported garbage. So I resolved to create a set of "custom" entities to handle with the XML filter, and used the translation grid filter of memoQ to make an inventory of these.

Filtering for source text content in the memoQ translation grid

That's when I noticed that the translatable data in this crappy faux XML file was actually HTML text. So I thought perhaps the cascading filters feature of memoQ might help.

Using all the defaults, I found that the HTML was fixed nicely with tags, but I did not want the tags that were created for non-breaking spaces ():

So I had another look at the settings of the cascaded HTML filter:

I noticed that if the option to import non-breaking spaces as entities is unmarked (it is selected by default), these are imported quite properly:

 

Now the text of some 600 lines was much easier to work with - ordinary readable document with a few protected HTML tags.

I'll be the first one to admit that the solution here is not obvious; in fact, apparently one of my favorite Kilgray experts took a very different and complex path using external tools that I simply understand. There are many ways to skin a cat, most of them painful - at least for the cat.

As I go through and update various sections of my memoQ tips guide, I'll probably expand the chapters on cascading filters and XML to discuss this case. But I haven't quite figured out a simple way to prepare the average user for dealing with a situation like this where the problem is not obvious. One thing is clear however - it pays to look at the whole file in order to recognize where a different approach may be called for.

Maybe a decision matrix or tree would do the trick, but probably not for many people. In this case the file did not have a well-designed XML structure, and that contributed to the confusion. My colleague is an experienced translator with good research skills, and he scoured the memoQ Help and the Kilgray Knowledgebase in vain for guidance. Our work as translators poses many challenges. Some of these are old, familiar ones, repackaged in new and confusing ways, as in this case. So we must learn to look beyond mere features and instead observe carefully what we are confronted, using that wit which distinguishes us from the dumb machines which the dummies fear might replace them.