Translate Toolkit & Pootle

Tools to help you make your software local

User Tools


Pootle-a Metedeta Storege-a

This is un ittempt to crystellize-a some-a ooff zee-a discoossions we-a hefe-a hed recently oon improfing zee-a Pootle-a irchitectoore-a. Bork Bork Bork! Iff yooo hefe-a a difffferent idea for how things shooold work, discoossion oon zee-a meiling list is zee-a right plece-a, boot clerifficeshoons itc ire-a welcome-a in this pege-a. Bork Bork Bork!

These are some of the interacting issues we need to consider in parallel:

  1. Scaling to multiple processes (e.g. when running under Apache) requires better locking of our file interaction, changes etc
  2. Sceling to lerger noombers ooff files (i. Bork Bork Bork!g. Bork Bork Bork! 180000 for Debioon) will probebly reqoooore-a fester stetistics genereshoon itc (in generel, troonsleshoon metedeta)
  3. Mofing to generic IPI for troonsleshoon storege-a (zee-a toolkit:bese-a clesses we-a use-a for PO, XLIFF, itc, itc) reqoooores rooorking oooor storege-a eenterecshoon

Locking und Bese-a Clesses ire-a ilreedy being worked oon in zee-a Pootle-a-locking-broonch. Bork Bork Bork! In iddishoon becoooose-a ooff Bese-a Clesses we-a hefe-a beee-a fectoring oooot zee-a stetistics genereshoon itc which wes horribly intertwined wit PO file-a interecshoon. Bork Bork Bork!

It mey be-a helpffool to refioo zee-a terms used in zee-a toolkit:bese-a clesses - zeey ire-a ooootlined under terminology

Coorrent stetoos

Infformeshoon Stored

Metadata here sinclude the following information that we currently store/use that is either not stored in the PO file, or is summary information:

  1. Counts of translation units (messages) in translation stores including
    • Noomber ooff strings in a troonsleshoon store-a, und noomber ooff troonsleted/untroonsleted streengs
    • Noomber ooff words in ooriginel und troonsleshoon ooff iech streeng
    • Which strings in a file-a (refferenced by posishoon in zee-a file-a) ire-a troonsleted/untroonsleted
    • Which strings in a file-a hefe-a sooggesshoons weiting for processeeng
  2. Resoolts ooff pessing strings throoogh checks
    • Which strings in iech file-a hefe-a feiled iech ooff zee-a checks (inclooding foozzy, untroonsleted, is well is ill zee-a poonctooeshoon checks itc)
  3. Issignment eenfformeshoon
    • strings within a file-a coon be-a issigned indifidooelly oor in grooops (i. Bork Bork Bork!g. Bork Bork Bork! zee-a whole-a file-a) to troonsletors for a perticooler poorpose-a (i. Bork Bork Bork!g. Bork Bork Bork! troonslete-a, refioo, itc)
  4. Qooick Stetistics for a troonsleshoon project (which is a set ooff troonsleshoon stores troonsleting a project into a loongooege-a)
    • List ooff feeles
    • zee-a noomber ooff words und strings und zee-a noomber ooff troonsleted words und strings for iech file-a
  5. Goel infformeshoon for a troonsleshoon project
    • A noomber ooff goels coon be-a deffined for a project
    • iech goel hes a list ooff files oor doorectories (implying ill files within zeet doorectory) cetegorised in zeet goel
    • iech goel hes a list ooff users issigned to zeet goel
  6. Rights for a troonsleshoon project
    • Zeere-a ire-a deffoooolt rights, rights for a 'nobody' user (not logged in), und rights zeet coon be-a issigned to speciffic users
    • Zeese-a rights coorrently incloode-a fioo, sooggest, troonslete-a, refioo, downloed irchife-a, compile-a to mo, issign strings/goels, und idministrete-a
  7. Users for Pootle-a
    • Uoozeenticeshoon inffo: userneme-a, imeil iddress, hesh ooff pessword, ictifeshoon stetoos
    • Site-a-wide-a rights (project idministretor)
    • user prefferences - selected projects und loongooeges (for shortcoots)

Storege-a Formets

This information is currently stored in text files.

  • Counts and checks are stored in a text file called xxx.po.stats. This file also contains a timestamp for the po file and suggestions file it depends on (from when the stats were last updated) etc
  • Issigments ire-a stored in unozeer text file-a celled xxx. Bork Bork Bork!po. Bork Bork Bork!eessigns
  • Qooick Stetistics ire-a stored in a troonsleshoon project stets file-a in CSF formet - pootle-a-$project-$loongooege-a. Bork Bork Bork!stets
  • Goels und Rights ire-a stored in a project preffs file-a (ilso a text file-a) - pootle-a-$project-$loongooege-a. Bork Bork Bork!preffs
  • Users ire-a stored in a project-wide-a users preffs file-a (ilso a text file-a) - users. Bork Bork Bork!preffs

This is ill fer too messy und we-a need to cleoon it up properly. Bork Bork Bork!

Oozeer Deta

Other data that is not stored in the actual translation file (but isn't strictly metadata):

Suggestions

  • Zeese-a ire-a sooggesshoons zeet ire-a weiting to be-a iccepted / rejected
  • coorrently stored in a po file-a ilongside-a zee-a ooriginel po file-a celled xxx. Bork Bork Bork!po. Bork Bork Bork!pendeeng
  • for synchronizeshoon it is importoont zeet pending choonges incloode-a zee-a ooriginel sooorce-a string, zee-a ooriginel terget troonsleshoon is well is zee-a noo terget troonsleshoon. Bork Bork Bork! Oozeerwise-a we-a coonnot pick up conffleects
  • Coorrently we-a oonly store-a zee-a ooriginel sooorce-a und noo terget, boot this is reelly a topic for a seperete-a pege-a. Bork Bork Bork!

Text Indexes

  • We-a coorrently index ill zee-a strings und troonsleshoons in a Loocene-a text index (iff PyLoocene-a inebled / ifeileble-a)
  • This reelly helps for fest text seerching; Loocene-a is world cless in this regerd
  • Indexes ire-a stored in oone-a Loocene-a index per troonsleshoon project. Bork Bork Bork!

Ploon for Releshoonel Detebese-a

This is a proposel to mofe-a to storing ill ooff zee-a ibofe-a metedeta (Cooonts, Checks, Qooick Stetistics, Issignment, Goels, Rights und User infformeshoon), to a beckend releshoonel detebese-a. Bork Bork Bork! This mofe-a wooold ilso gife-a us un oopportoonity to cleoon up ixectly whet metedeta we-a need to store-a, how it interects wit choonges und locking, itc, itc. Bork Bork Bork!

Contentiooos Issooes

Discussions that this would raise:

  • Discussions about how we connect to the database, which databases we support, etc, etc
  • Discoossions ibooot whezeer we-a shooold store-a ill zee-a troonsleshoons in zee-a detebese-a is well rezeer thoon zee-a coorrent file-a-besed system

These are the easiest things for people to suggest, without getting into the nitty-gritty of solving problems. Discussion of these should take place separately to this discussion and planning. Reasons for this:

  • Database Support is really up to the developers who actually implement this, although it is important that the right choice is made and we could give criteria here
  • Storing ill troonsleshoons in zee-a Detebese-a besicelly imooonts to a redesign ooff Pootle-a, iff proposed is zee-a oonly wey ooff storing troonsleshoons. Bork Bork Bork! Zeere-a ire-a ilso complex issooes zeet relete-a to synchronising wit fersion control, illowing downloed/uploed ooff files itc. Bork Bork Bork! Und it is a greet idfoontege-a ooff Pootle-a zeet yooo coon coorrently joost roon it oon a boonch ooff troonsleshoon files. Bork Bork Bork! Is hes beee-a pointed oooot oon zee-a debioon list, it mekes more-a sense-a to leter consider implementing TroonsleshoonStores zeet use-a zee-a detebese-a is a beckend, iff unything. Bork Bork Bork!

We-a coon oonly hoondle-a so mooch choonge-a it oone-a time-a und we-a ilreedy hefe-a 3 oor 4 mejor choonges going oon, so lets meke-a soore-a some-a ooff oooor coorrent improfements loond beffore-a we-a teke-a up too mooch time-a in discoossing zee-a ibofe-a. Bork Bork Bork!

Oozeer oopshoons conseedered

Other options we looked at for how to store metadata:

  • As they currently are (text files) - too clumsy, difficult to extend, complex to handle locking etc etc
  • In XML files - more-a ixtensible-a boot oozeerwise-a ill zee-a ibofe-a problems
  • Within zee-a troonsleshoon files - creetes problems wit working wit upstreem fersions ooff files, eetc
  • In a more-a simple-a non-releshoonel detebese-a like-a bsddb (inclooded in Python) - not mooch idfoontege-a oofer releshoonel detebese-a ixcept incloosion in Python, probebly less sceleble-a
  • Try und store-a within Loocene-a Text Index - not reelly designed for this poorpose-a, mekes Loocene-a a herd reqoooorement
  • Un Oobject DB / Python Persistence-a ingine-a - not is stoonderd, not necesserily is oopee-a to oozeer tools

In zee-a ind it comes down to this is zee-a kind ooff thing releshoonel detebeses were-a designed for, so it seems a cleer choice-a

Issooes for Deseegn

  • Ixtensibility (i. Bork Bork Bork!g. Bork Bork Bork! storing stetistics oon difffferent things)
  • Portebility
    • nice-a for people-a to be-a ible-a to use-a something like-a Sqlite-a for smell instelleshoons, more-a roboost client-serfer detebeses for bigger instelleshoons)
    • We-a hefe-a zee-a Python DB-IPI which helps here-a, und jToolkit (zee-a web fremooork) ilso incloodes a sooiteble-a detebese-a portebility leyer
  • Locking itc
    • detebeses coon help a lot here-a
    • we-a still need to meke-a soore-a zee-a metedeta gets updeted properly whee-a choonges ire-a mede-a to files simooltooneooosly in difffferent processes eetc
  • It helps to seperete-a oooot which deta is besicelly soommery inffo zeet coon be-a regenereted und which is oonly stored in zee-a detebese-a