Documenting auto-translation rule development for memoQ

In an recent article, I described my simple method of recording examples of structured information like dates, financial expressions or legal references to help developers plan auto-translation rules (or other features using regular expressions, such as Regex Tagger rules) in memoQ and other applications. These are a sort of simplified performance specification - a table of examples showing how the rules should "perform", what they should do: what patterned source language expressions are to be transformed into particular structured expressions in the target language.

The need for proper documentation of such efforts does not end there, however. It is very important, especially for more complex sets of rules, that there be clear documentation of the purpose and logic of the rules developed, and that this documentation be present

• in the rules themselves (as comments) and
• in external documents to be used as references for troubleshooting, maintenance and further development.

Auto-translation rules and other resources using regular expressions should not be scripted and maintained for the long run in memoQ itself or in any other environment which does not allow thorough commenting of the regular expressions used. Without comments, it is simply too easy to destroy functioning rules by forgetting why they were written a certain way once-upon-a-time, and an environment able to use comments also allows old rules to be "commented out" (disabled, but still available for reference or later re-use) while new versions are tested. That is basically impossible with memoQ's internal resource editors at the present time. And to make matters worse, if auto-translation rules are edited inside memoQ, their order changes, sometimes with dire consequences if functionality depends on the rule order. Try sorting out problems like that in a set of 70 or so rules.

Excerpt from a large set of currency format rules with extensive comments. These comments are stripped when
the rules are imported into memoQ, so all maintenance should be done externally in a tool like Notepad++.

As I began to revise and improve old rules that I created years ago for dates and currency expressions, I found that it was helpful to create a record of what changes I had made - and why I made them - and keep this information in a tabular form for easy reference and re-use.

Click to access a PDF sample of my rule development record (2 pages)

The graphic above is one example of how I maintain my personal records of some work developing regular expressions. I usually include

• descriptions of all information recorded
• a specific example on which I will base the general rule
• a simple ("fragile") version of the rule part (source input and target output) with only the most essential elements; this is not error-tolerant, but it is the easiest to understand and the first place to look if something isn't working as I would like it to
• more robust variations which take into account differences in spacing, punctuation, etc. or include things like non-breaking spaces that might be desired in the output (this can get cluttered and hard to read)
• color-marking for easier identification of some elements
• comments about why things are written as they are or about possible improvements or problems

This record is a template of sorts from which rules can be assembled very quickly or rules can be re-purposed for other languages or formats in a way that is easy to follow and catch mistakes. Such records are also helpful if the rules are to be shared with other developers or maintained by someone else.

My example is certainly not the final word in project documentation for such efforts; it is simply part of a set of personal tools to help me work more efficiently with the limited time I have. Professional development and consulting organizations often have far more extensive and detailed systems of project documentation; when I was part of one such shop nearly 20 years ago, my (downloadable) 2-page example might easily have filled twenty pages of very important-looking professional technobabble. Life's too short for shit like that anymore.

But if you value your time as a developer or your investment as one who hires others to develop such useful rules, it pays big dividends in most cases to demand some sort of clear, systematic and accurate record of how your special rules, filters, etc. were developed so that they can be maintained and improved in the future.

"Upgrading" translation memories for document context

Translation memories are in a way the Zombie Apocalypse of our profession. Dead data walking, with rotting bits falling off and lying out of context in a concordance for the reader to puzzle over, wondering where that bit of wordflesh one fit in the whole.

How often in my Dark Days as a Trados User did I look at some concordance hit and wonder just how stoned the translator was, corrected the "mistake" and discovered later it was quite right in context? Context matters, but with translation memories it's like the Invisible Hand at best, with a few missing fingers.

I once was lost, but now I found memoQ LiveDocs, and I

1. export my context-free or -invisible TMs to TMX, then
2. import the TMX to LiveDocs

where I now can go directly from a concordance hit to the translation unit with LiveDocs, which can be read in document context if that chunk of text was in fact written to the TM in the sequential order of the document, which is often the case.

The state of the upgrade: memoQ 2013 R2

The memoQ 2013 release started off on the wrong foot with me in many ways. I was deeply disappointed by the features that were previewed in Budapest at the last memoQfest, and I was even less happy after I saw what a hash had been made of one of the features I use most: comments. In fact, I wrote a rather annoyed blog post about that not long after the release. There was a lot of talk about "game-changing innovation", but frankly I really could not see it. My translating colleagues asked me if it was worth it to upgrade, and aside from my usual warnings about the need to wait for at least 2 or 3 months after any release for it to mature and stabilize, I just could not find any compelling arguments for a freelance translator to move from the stable, excellent 6.2 version to the rather dodgy 6.5 version, or "memoQ 2013" as it was rechristened.

Almost on the usual schedule, however, two months later the bugs were largely sorted out, the initial mistakes in the comment feature redesign were well fixed, and I no longer saw the memoQ 2013 release in the same dim light, but could actually see some benefits for my freelance colleagues to upgrade to that version and no actual harm in doing so. And as I got to know the fuzzy term matching feature better and saw how it helped me deal with typo-laden source documents or the usual spelling chaos of German technical writers, I began to see some very compelling value in memoQ 2013 for translators.

Most of the "game changers" talked about in May actually arrived a month ago with Release 2 of memoQ 2013. I did my best to lower expectations for this release, not because I think it is crap, but because I think this is one of the best CAT tool version upgrades I have seen in 13 years, and I knew it would need the usual time to mature. I think by the end of the year this version will have so much to offer that I would rather not have people stressing over the small stuff that I am confident will be fixed well.

However, I decided to live dangerously, and I switched over all my production work to use this version even before the official release. The first few weeks were not fun with all the little quirks I discovered and duly reported, but I encountered nothing data-destroying or really shocking, mostly just housekeeping details like somebody forgetting to vacuum the rug after gutting the whole house and giving it a nice remodel.

One month after the official release, memoQ 2013 R2 is far more reliable than I remember any memoQ version being one month after release. There has been steady refinement in its features, and I continue to discover hidden gems that I sometimes suspect most of the Kilgray team aren't even aware of yet because so much was added and changed, but not in a way that disrupted older work processes. I have a long shopping list of refinements that I think should be made to new features like the TM search tool (which has only actually worked on my system since the release of the 6.8.5 build about a week ago) or that ground-breaking monolingual review feature which (will probably be the next big CAT feature to copy), but even the new features I consider rather immature are already looking pretty damned good. I can't guarantee that this release can be trusted for all your work right now (though it actually seems pretty good to me right now), but since it can be safely installed in parallel with older versions, I definitely recommend taking a look and joining the conversation on refinements still needed. I think Kilgray has been very responsive to user feedback in this round, and I can't say I am anything but encouraged by what I have seen in the last month.

One very exciting change for me in the current build (6.8.6) is that the rather risky non-optional export of target text comments with DOCX files has been sorted out very nicely. The solution seems a little strange to me right now, but it's a great step forward with some excellent possibilities.

When I saw those "severity levels" added to the commenting features in memoQ 2013 (6.5), I had very little good to say about them. I still don't think much about how they are named and wish I could choose my own labels, but now I can only applaud their usefulness. Why? Because the addition of the five checkboxes above has given me the control I want over comments to be included in an exported translation of a DOCX file. I can cleanly separate the comments which are notes to myself from those for my project partners and comments for my customers. This is very helpful.

I do think it is odd that this control was placed at Tools > Options > Miscellaneous > Translation when the comment exports (as far as I know) only affect DOCX files, but if there are plans to extend this feature to other exported formats, then this makes sense. I would like to see similar filtering controls for the ordinary view filters (on that last tab where comment and tag filtering criteria can be specified) and for comment inclusion in a bilingual RTF export. Either of these would be an enormous help to my frequent work processes, because I use a lot of comments intended for different people, and sorting these out cleanly can be laborious.

In recent weeks I have been working on the new edition of my memoQ tips book and taking a very close look at "corners" of the software that I suspect very few have time or inclination to look in. And I've had days when it really felt like Christmas has come early. One discovery after another of nice little refinements, lots of incremental improvements, which added together give a total with what I feel is a lot of value. I'm writing way too many private thank-yous to some of the people at Kilgray for what I see as excellent new directions even if I am inclined to argue over some of the details.

Since the release of memoQ 6.2 and its follow-ups with the bilingual text/Excel filter, there has been such a steady flow of useful improvements to help individual translators work better that those who claim that all the effort of development has been spent catering to the corporate sausage-making interests of the low-paying cattle call crowd simply haven't been paying attention. Or they have been confused by Kilgray's occasionally appalling failure to organize their messages properly for different interest groups. If you're talking to a big group of freelance translators and start discussing "great server features to monitor your translators' productivity", don't expect blown kisses and showers of rose petals. Sometimes it's obvious that the makers of the tool don't always understand the importance of what they have created for our work. Well, why should they? We're the ones doing it. But I tell you, right now there is a lot more gold for individual translators in the memoQ mine than anyone realizes. That goes for me too. I am surprised by fat new nuggets I find almost every week.

Do I care that so much effort is spent on developing cutting edge project management features for memoQ translation servers, even ones that I think can be abused in some pretty awful ways by some companies whose business practices I detest? Well yes I do... I think it's great. Besides, I can actually come up with nice uses of those awful features. You can do a lot of things with a cutting-edge: chop up a tasty salad... or the local nursery school. Blame the fool, not the tool.

Kilgray has avoided the disastrous errors committed by Atril in the last decade as their market mis-focus and disastrous failure to get the maintenance revenue needed to fix and develop features steadily eroded the ability of its loyal users to cope with a changing market. There was nearly a complete failure to compete for the business of translation agencies and corporate and government translation departments. And the solutions that prevailed in those quarters were mostly rather awful. I watched whole departments of Siemens traumatized by the disastrous Trados Teamworks, which made a number of those in the translation team of the medical products division look forward to retirement.

Kilgray has steadily built its business in the markets ignored by Atril a decade ago and in doing so has secured its future far better and ensured the funding of a truly remarkable series of improvements in the four and a half years I have been using memoQ. And now... when I look at the features of the recent SDL Trados 2014 release I see good things that I have known from other tools for a long time for the most part, nice to have really, but as I stifle a yawn I wonder if it all really has to be so complex since I'm not depending on consulting or training for SDL to pay my bills. And then I get back to memoQ and keep getting rocked by the "wow factor" as I find useful new things while trying to concentrate and get a job done. memoQ 2013 R2 is one of the worst offenders I've seen in a long time for its very real threats to make my work a lot easier and more fun!

memoQ filters for static and dynamic views and navigation

The filtering functions for translation documents in memoQ are really cool. I'm not talking about the import filters for different types of documents, though most of these are rather good, and improvements are being made all the time. I mean the ways in which you can use filters to look at the content you are translating and sort and navigate it in different ways.

The filters can be used to create static views from one or more translation documents selected on the Documents tab of the Translations window of Project home using the Create view command.

I use this function a lot to create comment and feedback lists for clients or select some particular part of my content that I want to save and work on or share separate from the rest. Exported as bilingual RTF tables, the content can be corrected or questions answered in the comments column, and all the changes and commentary can be re-imported to update your project.

What I use even more often are the dynamic filters in the translation window. There are three main types: sorting filters in a dropdown list, source and target text filters in the fields above the related columns, and the dilog filter with its many options, which is invoked with the funnel icon (2):

Source and target text filtering of the segments can be made case-sensitive by marking the icon (1). Any number of filtering operations can be applied cumulatively in sequence, and filters applied with the source and target text fields can be cleared with the red X icon (4). To clear a sorting filter, you must select "No sorting" from the dropdown menu.

These view filters can be very helpful for translation and quality assurance. But what many do not realize is that memoQ also allows you to navigate through translation segments using filter criteria. This is done with Edit > Goto Next (Ctrl+G). The filter criteria to apply for navigation are chosen under Edit > Goto Next Settings (Ctrl+Shift+G).

This often has the advantage over view filtering that all the segments remain visible and you can see the context better. Examples of this are shown for navigating to commented segments and navigating through the many footnotes in a document to check their formatting in the short (3.5 minute) video tutorial below. It was prepared with the most recent build of memoQ 2013 (6.5.15) and shows the new "golden" bubble icon for commented segments. The video demonstrates (with footnotes) how tag type can be used as a filtering or segment navigation criterion. This might come in handy for an academic thesis or a legal document with many footnotes to check.


Use the icon at the bottom right of the video to toggle full screen mode for viewing; this makes it much easier to see the details of this somewhat fast-paced clip.

0:17  Static views on the View tab of the Translations window
0:30  Dynamic source and target text field filtering
0:40  Dialog filtering with the funnel icon (3rd cumulative filter)
0:52  Using filters to navigate: Goto Next (rationale & contrast with view filter)
1:20  Goto Next settings for navigating commented segments
1:42  Navigating footnotes in a translation document with Goto Next
1:55  Setting the navigation filter for a tag type
2:30  Getting rid of a static view to allow segment joining
3:23  Oops! I join something I shouldn't and split the segment again, hoping nobody will notice.

Subscribe to my free YouTube channel and I think you'll receive updates of new video tutorials I add (or at least it'll be easier to find them). I would also like to thank Ulrich Scheffler of LSP.net for providing me with a Camtasia license recently to support my teaching - it's much better than the free Open Source CamStudio I started working with months ago, though I can definitely recommend CamStudio as a good tool to get started with making demonstration or teaching videos.

Comments on memoQ comments and YouTube playlists

I recently produced a small video tutorial on what I feel are the useful aspects of the comment feature in memoQ 2013. Although quite a few new things have been introduced to commenting in the current version of the software, the real significance of these changes for ordinary users of the software is limited. Now that what was broken in the memoQ 2013 release is largely fixed, those who care about comments for offline use can continue to use this great feature without much inconvenience.

Here is an idiosyncratic overview of how I use comments in my projects (HINT: these embedded videos are easier to watch if you do that in full screen mode by clicking the icon at the lower right of the play window):

Time  Description
0:28  Opening the comment dialog
1:01  Commenting highlighted text
1:48  Adding "codes" to comments for later filtering
2:44  Selecting all files, creating a view with all comments
3:35  Comments shown in speech bubble tooltips
3:57  Creating a filtered list of comments (code = '@PM')
4:50  Creating a filtered list of comments (code = '@CST')
5:20  Exporting commented segments in a bilingual RTF file
6:15  Check segments for extraneous comments before sharing the exported list

What I didn't show here is my usual way of accessing and exiting the comment dialog: keyboard control, opening with Ctrl+M and exiting with a quick tab to the OK button and hitting the Enter key. Having multiple comments makes editing slightly less convenient if one has to click on an icon, but the ease of deleting an entire comment in a series, and the separation of comments by a new paragraph in an exported RTF bilingual file are compensating conveniences.

That six-and-a-half minute video really has more information than someone generally familiar with the old way of using comments in memoQ would care about. The only part which might really interest someone who already knows how to create an exportable view with commented segments is how the procedure for creating views of selected comments differs from creating a view with all comments. So I decided to use the excerpting feature of YouTube playlists to create a special "view" of the tutorial video which shows only that little bit from which I believe many experienced users may benefit.

(Use the link above to look at the trimmed playlist on YouTube - I removed the embedded video code here, because its behavior in the Google Blogger environment seems to be quirkier than links on a Moodle web page or Facebook page. This technique is useful but may still require careful testing of the environment in which it will be used.)

No index needed here - the video is barely over a minute long in its two parts. This technique of playlist excerpting on YouTube could be used to "mine" longer teaching videos for specific bits of information needed to understand a specific issue. One can combine separate video clips, "in whole or in part" as contract lawyers like to say, or individual segments of a single video as I have done here. This is a useful technique which, along with time index lists such as that shown above, I hope to see applied more often for the education and support of translators.

How does Kilgray present the state of commenting in memoQ? The video clip below is from Kilgray's YouTube channel - interesting, but really another world. The video shows the commenting feature as it was at the end of May, with "innovations" which sparked the Commentgate controversy.


This presentation is really very focused on users of the memoQ server, because all that lovely highlighting is only visible in the memoQ environment itself, and these comments with highlighting do not currently export in the usual medium for sharing feedback (comments) with clients offline: RTF bilingual files. In fact, it's really a shame that in all the years that memoQ has offered exportable comments, this very helpful feature has hardly been part of the official teaching, because in the real world of client relationships, it is often a great asset.

Innovation? No comment. A black mark for memoQ 2013.

Update August 2013: problem solved.

"This comment thing is the last straw. I am definitely not upgrading to the new version of memoQ!"

The response from a friend who asked me to show how the comment function in memoQ had devolved in the latest version (memoQ 2013, aka version 6.5) revealed the frustration of a user whose main interest in new product features for much of the past year has been how to disable or avoid them. Unfortunately for her, there appears to be no escape from the latest innovation, which one member of the memoQ Yahoogroups list suggested would become known as Commentgate. This may be a good example of the old adage "if it ain't broke, don't fix it".

The old comment function has been at the heart of my use of memoQ for years, and the ability to export comments to share with my clients was one of the main reasons I pushed Kilgray to introduce the bilingual RTF table exports which are so popular for editing and translation. Comments added in a word processor (such as Microsoft Word) could be re-imported to the memoQ project and reviewed. It was all very quick and simple.

The old memoQ comment dialog with a personal note on edits required

If the comments included questions about a text or a term, the response in the comment column of the RTF table would be included when the bilingual file was re-imported, which was often convenient for editing purposes. A record of questions and answers could be maintained fairly easily in the editable comment field.

All that has changed with memoQ 2013. Disastrously so in the initial release on May 31. In their eagerness to implement an LQA quality assurance model relevant to only a minority of users, mostly the sort of agencies who prefer metrics in place of actual quality, Kilgray dynamited the previous straightforward, robust comment function, adding dropdown selection fields for "severity level" and scope.

That wouldn't be quite so bad despite the extra steps needed to add a correctly classified comment now if it were possible to edit the comments. It is not possible to edit comments in memoQ 2013 in the current release. Nor are all the comments included in an RTF table export. Only the last comment is included; all others are lost:

If a comment is altered in the bilingual RTF file, when the bilingual is re-imported to the project, a new comment is created and classified as "Information" applicable to the entire row:

This is all really a shame. In the effort to push server-based workflows and cater to a limited special interest group, memoQ's architects have managed to sabotage one of the tools (or two, depending on how you count) which have contributed to their great success in recent years. And unfortunately, unlike other recent, often irritating "innovations", such as the various target autotext options that drive many users of dictation software batty, this new type of comment can't be switched off so that we can work in the old, accustomed way.

Many users have already objected strenuously to this broken functionality, and some compromise solutions have been suggested by Kilgray. One of these, which involves a delimited export of all the comments in the RTF bilingual export, would be reasonable. Whatever changes are made to commentary in memoQ 2013, I hope this will include making comments editable very soon with appropriate rights.

The two selection field in the new comments dialog have a default behavior that is probably useful in some cases. Once a comment has been created, the next comment made in the text will assume the same "severity" rating applies (not necessarily a valid assumption, but if I am going through a text marking things of the same type this can be useful). All comments are assumed to apply to the target text by default. This is a bit of a nuisance to me, as most of the comments I make in a file refer to errors or unclear expressions in the source text. But really, for the way I work, the two classification steps with the dropdown menus are just extra work and additional sources of possible errors and/or confusion, so I would be quite happy to bypass these altogether.

I do like the idea of a comment history for a text. This would be relevant and useful for the way I work. But overall, the current implementation of the comment function in memoQ 2013 does not serve my interests at all and creates unnecessary complications for me and many other users.

Spamalot: desperate translation agencies

Soon after I started this blog in 2008 I found it necessary to take various measures to protect against spam comments. In the early days, the spam was a lot of the same that somehow makes its way to my e-mail inbox, usually real estate scams, offers of wondrous herbal mixtures to supersize various anatomical parts or God-knows-what in Chinese. Spam filters have become sophisticated enough to catch most of that now, so that the trash encountered most of the time now appears to have a bit more human intelligence behind it. That is, if you associate bottom-feeding translation agencies with human intelligence in some misguided way.

I have a definite impression that the competition in the race to the bottom for low prices and garbage quality has led some of these agencies to think they can improve their position in the global dog race by getting backlinks on popular translation blogs. This is done in comments on blog posts, sometimes as a link embedded in the rather vacuous text (as Rosetta once did to a New York agency rival's blog below) or in the link associated with the poster's name (as "Cassy" - surely not its real name - tried above on behalf of "LanguageTran"). What started out as the practice of a few sleazy Pakistani agencies years ago seems to have become routine for a number of UK and US bottom-feeders.

I've seen this happen on almost every translation blog I read. There are so many attempts made on my blog that I make it a point to check the name link on every comment that seems a bit "vacant", and a very large number get flushed to where they belong.

I'm not against comments from agency owners or personnel - quite the contrary. These people are a legitimate part of our business and have a lot to contribute to discussions - and many do. I love it when they or others have something interesting to say in a post discussion, particularly if they can point out an error on my part or add some useful point. But if the point of the visit is simply to post drivel in the hopes of getting a link to drive traffic to your agency's site or improve its search engine ranking, y'all can save yourselves the trouble because you will most likely not be getting through.

memoQuickie: feedback comments

Exporting comments in memoQ in a bilingual table for others to comment further and correct, then re-importing to update translations and the TM is a very useful feature. I use it often to inform clients of uncertain terms or errors in the source text. You can too, as follows:

1. While translating, use Ctrl+M or double-click the speech balloon on the right to open the comment window and comment the segment.
2. On the Translations page of the Project home tab, select the document(s) with comments to export and click Create view. Choose the advanced options:
   
3. Then mark the checkbox for Commented on the Status tab:
    
4. To export the new view as an RTF table, select it and click Export bilingual and choose the Two-column RTF option. To refine your options, click Next, or just export the table.
    

The exported table shows only commented segments: