Wikipedia PDFs in scripts other than Latin

What normally goes wrong

Take a non-Latin article through most converters and you meet one of three failures, all of which look like the tool is broken and are really two separate problems.

The first is missing glyphs: rows of □, or blank space where characters should be, because the font in the file simply has no outline for them. The second is missing shaping: Arabic letters printed in their isolated forms instead of joined into words; Devanagari or Tamil vowel signs sitting after the consonant they belong in front of; Hebrew running the wrong way. This one is structural — a PDF’s text-drawing operators place glyphs at positions and have no notion of joining or reordering, so anything that does not do the work in advance cannot get it right.

The glyph half: 56,103 characters

Whatever font you choose is used for everything it can draw. Everything it cannot is filled in from 30 bundled faces — Noto for each script, DejaVu, a mathematics face, and a monochrome emoji face — which between them cover 56,103 characters. The CJK face alone accounts for 44,296 of those.

The filling happens per character, not per block, and only from a face whose own character map really has that character. That distinction is the whole of it: choosing a face because it suits the script, then discovering it lacks one character in ten, produces a document that is invisibly wrong — text you can select and copy, over blank paper.

If no bundled face has a character at all, you get a visible □ rather than a silent gap, and it still copies out as the real character. A mark you can see is worth more than a hole you cannot.

The shaping half, and which you get

Because PDF text operators cannot join or reorder, correctly shaped text has to be produced before it reaches the page. There are two ways to do that and they give you different files, so it is worth knowing which one your article took.

• Real text Selectable, searchable, copy-pasteable, with working links. Latin, Greek, Cyrillic and CJK always. The Indic and other left-to-right complex scripts — Devanagari, Bengali, Tamil, Telugu, Kannada, Malayalam, Sinhala, Thai, Khmer, Myanmar and the rest — whenever the document converts cleanly.
• Images Each line drawn as a one-bit stencil after the browser has shaped it. The glyphs, the joining and the direction are right and it prints perfectly — but that text is not selectable or searchable. This is what Arabic, Hebrew, Syriac and Thaana get.

The choice is made per document rather than per line, so a document is never half one and half the other. For a left-to-right complex script the conversion has to come out clean across essentially the whole article — if it does not, the document falls back to images entirely rather than shipping a few mangled paragraphs among good ones.

There is a control if you want it, under PDF Options: Automatic, Images, or Real text (force). Automatic is the default and is right almost always.

Right-to-left documents read right to left

Direction is decided per block, from the article itself, so an Arabic article with an English quotation in it sets both correctly rather than forcing one on the other. In a right-to-left document the whole page follows: text is aligned to the right, and a gallery of pictures fills from the right, so the reading order of the plates matches the reading order of the text.

Typography that knows the language

Justified text is hyphenated using the same Liang/TeX pattern data TeX uses, and the app ships pattern sets for 48 languages — most of the Latin, Cyrillic, Greek, Armenian and Georgian ones for which patterns exist. Words break at real syllable points. A language with no pattern set gets no hyphens and nothing else changes.

Chinese, Japanese and Korean are handled differently on purpose: a CJK-dominant line is not stretched at its spaces, because there are none to stretch and forcing the measure would open gaps between characters instead.

Search works in the language too — pick the edition and type in its own script. You are searching that Wikipedia, not an English index of it.

What to expect

• Arabic, Hebrew, Syriac and Thaana are images. Correct on the page and in print; not selectable or searchable.
• A character nothing covers prints as □. Rare, and deliberately visible rather than blank.
• The PDFs are not tagged for screen readers in any language. The about page is explicit about this.

Everything above is the same in every one of the 168 editions — this is one pipeline, not a set of per-language special cases, which is why an article that mixes three scripts comes out right rather than right in patches. It applies equally to a book of several articles, which may take its chapters from different Wikipedias, and to a copy going on paper, where a rasterized script costs you nothing at all.