What's wrong with my segmentation (in translation)?

The fifth open office hours session for the self-guided online course "memoQuickies Resource Camp" discussed segmentation problems with documents imported to translation environments such as memoQ, Trados Studio, Phrase, Cafetran Espresso, etc. and various ways that these issues might be identified so that they can be corrected.

Segmentation problems waste enormous amounts of time, and bad segmentation rules are a plague on the translation and localization service community. Unfortunately, nearly all the rules I have seen, for all working environments, simply suck sewage. memoQ's rules usually suck less, but still....

This week's talk presented, among other things, some methods for identifying segmentation trouble spots quickly and easily with the use of special regular expressions describing common patterns followed by texts with troubled segmentation. And a Regex Assistant library has been provided (and will be updated during the course period) to help with all of this.

The video and related course pages will remain completely open to the public, with downloads available, at least through the end of 2023. After that the pages and resources may be taken down for updates and reorganization in other courses.

The video recording of the lecture "What's wrong with my segmentation?" can be accessed on YouTube (embedded below) or course participants can access the page to download it by clicking the "segmentation rules" icon at the top of this article.

An important part of checking the performance of your segmentation rules and possibly improving them is to have a good sampling of test data. One of my favorite sources for this are the European Community archives at the DGT, where EU legislation and other important information is available in a parallel corpus of all the official languages of the Community.

I have downloaded part of the 2022 DGT distribution and prepared a number of monolingual and bilingual corpora (about 2.6 million words, approximately 150,000 TUs) in EU languages and translation pairs. Moreover, information on my method has been published so that others can reproduce it for the languages that interest them.

The memoQ Regex Assistant Revisited at 15:00 CET on 21 September 2023!

 

Eleven months ago I was supposed to talk about terminology in a three-hour evening class taught by one of my friends at Universidade Nova de Lisboa, but I was so excited about the progress of the quality team I was training at one of my agency clients in Portugal that I twisted what was expected to be my usual straightforward 90 minute lecture on term base best practices in memoQ into an unusual take on the role regular expressions might play in terminology management.

That was a weird one, for sure, but the potential I saw was very real. I was defining "terminology" in rather broad terms to include not only more efficient term base management based on problem patterns but also translation memory clean-up, better filtering and find/replace operations in the working grid and QA.

"What the heck has all that got to do with scary ol' REGEX?" you're probably thinking.

Well, all this was triggered by memoQ's recent release of the One Ring Thing we've been needing to unify our memoQ management processes for the routine use of regex by The Rest of Us. The Regex Assistant. The Nazgul of Trados World are surely jealous.

That quality team I was training, my friends at Linguaemundi. is headed by Inês Lucas, about whom I had heard many good things for years from her enthusiastic professors at university but whom I hadn't actually met until she was hired by the agency a few years ago. At the recent memoQ Fest in Budapest, she explained how changing the approach to regex mastery from the struggles of syntax to organizing and applying packaged solutions in well-engineered processes significantly upgraded their work capacities and reduced stress levels. When you cut the nerdy crap and focus on understanding what solutions are called for particular tasks, everything gets much easier.

I was so amazed to see people who had struggled for years to learn regex well enough for simple tasks suddenly become solution powerhouses that I put together (rather spontaneously) a series of three online 90-minute workshops, which were repeated a month later. And new refinements to these methods come each time the ideas are presented.

The raw recordings of those six workshops are included in the current online course ("memoQuickies Resource Camp"), but one - the first of six - is publicly available on YouTube, where you can have a look.

However in the memoQuickies Resource Camp, a self-guided course that is serving as a platform for me to organize and distribute the best resources from my 14 years as a memoQ user, solution provider and trainer before I retire, I'll be taking another more streamlined pass at teaching some of the best possibilities for using memoQ Regex Assistant resource libraries. The webinars offered in most weeks of the course are simply an overview of the current topic emphasized in the course and also serve as a Q&A platform and a means of offering some different perspectives on information from the self-guided units. Recordings are always added to the course for later viewing.

This coming Thursday at 15:00 Central European Time, I'll give a brief overview of the Regex Assistant much like the public YouTube video does and answer any questions that attendees might have. Further information and an event notice can be found here on LinkedIn.

You can join the webinar with this link. The meeting ID is 878 3540 2561, and the passcode is 385434

New online course: "memoQuickies Resource Camp"

Summer is almost over, but technically, "camping season" will continue in memoQ World until November 30th. Or maybe January 31st, depending on how you count.

Today, a three-month journey of exploration begins, covering six important kinds of resources to make work with the memoQ translation desktop and server environments more pleasant and efficient... and profitable. This self-guided online course will give participants full access to my 14 years of cumulative experience as a memoQ user translating, managing projects and developing hundreds of solutions with this world-leading productivity tool.

Click here or on the icon bar above to have a look at the course description and to see (and maybe download) some of the publicly available information and resources for better work in many language pairs. 

The emphasis of teaching will shift to a new resource every two weeks (with auto-translation rules as the main topic for the first two weeks), but throughout the course, information will be added continuously to all topic sections as I trawl through, sort, upgrade and publish the best or most interesting stuff from my archives. And course participants have access to open virtual office hours each week on Thursdays and some other occasions, where any questions can be asked and special requests made.

A special enrollment discount of 40% is available for the first week (code: HALFOFFLAUNCH) until September 10th, but you can join at any time and work with any of the material posted, ask questions and receive feedback. Learning material and downloadable, ready-to-use and -adapt resources will continue to be added until the end of November, and the full course will remain online through January 2024. Enrollment fees and content are subject to change without notice.

Addendum 1: On Thursday afternoon, September 7th, 2023, a presentation was made to introduce the first course topic - "Auto-translation Rules for Everyone". The recording and slides can be found here.

Addendum 2: Payment options for groups and monthly budgets have been introduced now. These options enable teams, departments and organizations to obtain blocks of passes for their members to receive continuing professional education in translation workflow tools. The host site applies VAT and other taxes where relevant and generates appropriate invoices. All relevant information can be found at the bottom of the information and enrollment page.

memoQ Regex Assistant workshops re-run

The series of three workshops on the use of regex resources in memoQ, with a particular emphasis on the integrated Regex Assistant library, has been updated and will be offered again on March 9, 16 and 23 from 3:00 pm to 4:30 Lisbon time (4:00 pm to 5:30 pm CET, 10:00 am-11:30 am EST).

You can register here to attend any or all of the three sessions:
https://us02web.zoom.us/meeting/register/tZEpde-sqTkvGtCdMsrBl825tFrpDQ98FkAI

This is an evolving course, with the content continuously adapted in response to new questions, workflow challenges and process research as well as interoperability studies with other tools. Participants in the last series asked quite a number of interesting things during and after the talks, and their questions provided excellent material for new examples and approaches, and I hope for the same experience in this round.

The memoQ Regex Assistant is a unique library tool introduced in its current form in memoQ version 9.9. The little bit of public discussion there has been about this tool is quite misleading. Contrary to the "pitch" from memoQ employees and nerdy fans in the user base, this isn't really a tool for learning regular expressions. There are much better means for doing that. And I have strong personal objections to the idiotic statements I hear so often that "everyone should learn some regex". What utter nonsense.

What everyone should do is take advantage of the power regular expressions offer to simplify time-consuming tasks of translation, review, quality assurance and more to ensure accuracy and consistency in language resources and translations. The Regex Assistant helps with this by providing a platform where useful "expressions" can be collected and organized with readable names, labels and descriptions in any language. These libraries can be sorted, exchange with other users and applied for filtering, find and replace operations, QA checks, segmentation improvements, structured translation of dates, currency expressions, bibliographic information, legal citations and more or exported and converted to formats for easy use in other tools such as Trados Studio, Phrase/Memsource, Transtools+ and more. All without the need to learn any regular expression syntax!

An exported Regex Assistant library converted to a readable format by XSLT

My objective is not to teach regex syntax. It is to empower users to take more control of their work environment and save time and frustration for their teams and enjoy more life beyond the wordface. To help with that, I provide some usable examples in a follow-up mail after each sessions: resources that you can use in your own work and share freely with colleagues. 

And in this next round of workshops, available for purchase, there will be some additional high value resources to help achieve better outcomes for work in particular language pairs and particular specialties, such as financial translations. These complex resources were developed over a period of years, sometimes at great cost. In the last session I'll be getting "down and dirty and a little nerdy" to show you my way of maintaining complex resources like these auto-translation rules and others in a very effective, sustainable way that enables you to adapt quickly to changing requirements and style guides.

Sign up free to join the fun here.

memoQ&A: The Regex Assistant in Practice

Note: there will be another series of workshops for this subject matter in March. Details are HERE. Register once to attend any or all of the sessions.

Users of memoQ version 9.9 or later have a powerful library tool available with which they can organize solutions or solution elements that use regular expressions and apply these without the complications of learning regex syntax. In each session, we'll look at different ways in which these portable libraries can be used, with a particular emphasis on solving common problems faced by translators and reviewers. Materials will also be made available to participants for later study and practice.

There will be three sessions of 90 minutes each on three consecutive Thursdays: January 19, January 26 and February 2 at 11:00 a.m. Lisbon time (i.e. noon CET). The first session will introduce the Regex Assistant library and its basic functions for organizing and exchanging information and then move on to specific examples of using the library to deal with common problems encountered in translation and review work. Particular emphasis in the first session will be on filtering and Find/Replace operations. 

The two later sessions will continue to explore filtering and options for making changes to texts and tags, and we will also take a tour of possibilities for using regular expression resources (from the library!) in other parts of memoQ such as the Regex Tagger, QA checks or auto-translation rules. As time permits, examples or requests from participants can also be explored. 

Those interested in joining the free sessions can register here.

Update: a recording of the first session is available here: https://youtu.be/KKR5aH5oGH8

To get a little taste of what's to come, have a look at this video created by a colleague last year:

Get better dates with memoQ

There's hope for all those incels stuck with RWS Trados Studio, Memsource, Wordfast, OmegaT and a host of other horrors if they are willing to make a change....

Not surprisingly, dates can be a real nuisance to translate and check, depending on the client's specifications. Target language specifications that include the use of elements such as non-breaking spaces can be particularly troublesome. But even apparently simple tasks like writing all dates in the target language as DDMonth-AbbreviatedYYYY or the like can go wrong far more than expected, and skilled reviewers easily get caught up in the flow of the text and overlook details of format (and often even correct content) in dates. This proved to be a shock to one LSP client who found hundreds of overlooked date errors in a large volume of recently reviewed text.

What's the solution to these inevitable human errors? Proper automation of the monkey work so professionals can concentrate on what they are good at: a fluent text that accurately reflects the intent of the original.

In the case of the QA horror of checking four source languages to ensure the dog's breakfast of date input formats (which also included day+month and month+year entries) regardless of capitalization, memoQ enabled a simple auto-translation ruleset to be created (a few hours' work, including testing and documentation); this was then attached to a project using a QA profile configured to check only against the enabled auto-translation rules, and BOOM! after about a minute, all the date errors in something like 100,000 translated words were revealed. The only false positives found were a few instances where times were written after the date, and the rule can be updated easily to avoid this issue.

I do a lot of date rule development for many languages, and I've published some of this in simplified forms on this blog. But interesting new tricks come up all the time. And I've found it useful when developing rules for others, who usually understand little about writing proper specifications that capture all the likely source input, to create special screening rules like the one shown in the first screenshot, which can be used to examine an entire large TM imported to the memoQ working grid, and see how the input and target texts vary. I used that expression on two large TMs in a view, and in just a few seconds, my laptop screen showed me all renderings of every English date in those TMs into Portuguese. While researching target formats for some new rules I also found quite a number of errors in the TM which I could have corrected had I cared to.

Having rules like this available in the translation phase can prevent quite a few errors to start with. I've found too many cases of dates in March translated as May and overlooked by both the translator and the reviewers. memoQ is - as far as I know - the only tool which will offer such conversions in a results table from which they can be inserted just like any other "terminology hit".

Other tools like RWS TradoZe Studio will allow you to use regular expressions for quality assurance (checking the text), but I'm not aware of any tool other than memoQ which allows you not only to include these checks in customized QA profiles but which can also provide them for on-the-fly review from a library of named expressions. That's what memoQ now does in version 9.8 (scheduled for release in June, within a few weeks) as shown in the first screenshot above.

This new rules library feature of memoQ makes it possible for the first time for users who have a life which does not involve wasting brain cells learning to program regular expressions to use the efforts of those who actually like that sort of thing and do it well. So with this tool, anyone can easily check for things like date errors and a lot more without knowing a single bit of regex syntax. That's some progress :-)

Stuff like dates just keeps getting better for memoQ users, leaving them more time for life and better stuff... like other kinds of dates.

If this kind of thing interests you, I think my friend Marek Pawelec may be teaching an in-person course in regular expressions for memoQ in July of this year, and he, I and others (including memoQ's Business Services unit) are available to help you with turnkey solutions to project challenges like those described here.