Troubleshooting encoding problems

In an HTML document, special characters can be represented by entities, either:

Those entities will need to be decoded to be displayed, or to process the content. For example the indexer doesn’t know what Français means, but it knows what Français does.

Note that the encoding (see below) of the document doesn’t matter in this case, because HTML entities are always expressed with ASCII characters (either numbers, or ASCII letters to compose entity names) so it doesn’t matter which encoding the document is in.

The first step is to look at the original content, as the crawler will see it, by making a simple HTTP request to the original web server.

With the old Squiz forums as use case, we can observe that it looks correct, we don’t see any corrupted characters in the output:

However, that doesn’t mean that the content is not problematic because browsers are usually smart (smarter than our crawler) and can automatically fix a number of encoding-related errors. Additionally, collecting more info on those corrupted characters will help us later.

The first thing to look at is the HTML source of the page, to see how those characters are represented:

In this case those characters are represented as HTML entities. That’s perfectly valid and the entities used there are valid too. Looking at the list of HTML entities:

This content seems ok.

Our use case seems to be related to processing entities, and so it’s probably not related to actual encoding problems on the original document. When that’s the case, other information is useful to lookup.

Depending on your Funnelback version, you’ll either have to use warc.pl (up to v13) or WarcCat (v14 and above).

Looking at the extracted content, we can see something interesting: The HTML numeric entity in the title has been converted to a named entity:

Example extracted content as html:

While surprising, that’s not necessarily unexpected as the filters can do that especially when parsing HTML with JSoup. In any case, that’s still valid HTML and valid entities.

Regarding the line breaks after Hi, the numeric entities seems to have disappeared from the HTML source.

However, if we fire up our trusty hex editor, we can actually see what happened behind the scenes.

HTML source:

Hex of HTML source:

Observe that what is between the opening and closing P tags is 0xC2 0xA0. This is the UTF-8 representation for the non breaking space, which was previously represented as the   named entity.

So here, we’ve actually found one problem: The page is still declared as ISO-8859-1, but non-breaking spaces are represented as a UTF-8 sequences. Because the declared encoding is ISO-8859-1 it is likely PADRE will index the document with this encoding. Instead of interpreting 0xC2 0xA0 as a single UTF-8 character (non-breaking space), it will interpret it as 2 separate ISO-8559-1 characters, with:

That explains why a  is showing up in the results!

It’s likely to be caused by a filter, because the crawler by itself doesn’t modify the content. The only way to fix that is to pinpoint which filter is causing the problem.

The first step is to determine which filter(s) are causing the corruption. This is done by editing the filter.classes parameter in the data source configuration, removing all the filters, and then adding them back one by one:

In the data source configuration:

You need to run an update between each change, to cause the content to be re-crawled and re-filtered.

In my case I found that MetaDataScraperFilter is causing the problem. It’s a Groovy filter living in lib/java/groovy.

Browsing the source code, we can see that this filter converts the HTML content into a JSoup object, to be able to manipulate the HTML. While doing so, it tries to detect the charset of the document:

Let’s print this charset by using a logger, and inspect the crawler.log (or crawler.central.log, crawler.inline-filtering.log for older versions).

jsoup filter example:

Charset is detected as Windows-1252, which is equivalent to ISO-8859-1 for our purposes (see the Indexer section below for further explanation), so that looks correct.

However, despite the charset detection being correct the content is still read wrong. That’s because of:

The call to input.getBytes() used to convert the content string to an array of bytes doesn’t specify a charset so it will use the default one as said in the Javadoc. The default encoding in Funnelback is UTF-8. It means that the detected charset of the content will be Windows-1252 but the byte stream will be read as UTF-8 resulting in corruption.

This corruption is only visible when the string is written back, making the problem harder to diagnose.

One should always be careful when converting Strings to byte arrays String.getBytes() and vice-versa new String(byte[] data. If the charset is not specified it will use a default which might not necessary be what you want. (Usually UTF-8, but that’s platform dependent unless specified with a command line argument when starting the JVM). It’s better to always specify the charset to avoid any problems:

This specific code is not easy to fix because an InputStream is needed to detect the charset, but you need the charset to create the InputStream from the String! A better way to do it is to build the JSoup object from the input string itself. This way you need not to worry about providing an encoding with an InputStream .

The content reading is wrong here, but for this guide sake let’s inspect how the content is written back:

It’s simply using the JSoup Document.html() method to do so. We need to dig into the JSoup documentation to understand what charset will be used by this method. By doing so, we find the Document.OutputSettings class.

Let’s add some code to inspect the charset from the output settings just before writing the document:

Inspecting the crawler.central.log:

That’s the correct charset, but we can still confirm that something is wrong in the filter by outputting the content before and after filtering, and compare both:

Before:

After:

There’s no easy way to automagically find the encoding of a file so it’s better to know what encoding was used by the custom gather component. However, you can get some clues by looking at the content itself.

Ensuring your workflow script reads and writes it right

This is highly dependent of the language used to implement workflow scripts. Usually API calls to read or write files will allow you to specify an encoding.

When using Groovy there are different ways to configure the encoding:

These options need to be used carefully as they will completely change how PADRE interprets the content:

If a document has no META tags or HTTP headers to give a clue about it’s encoding, PADRE will default to Windows-1252. You can use:

In practice, those options should only be used as workaround if you can’t fix the root cause of the problem, rather than being attempted in the hope of magically fixing the problem.

First step is to use padre-sw.cgi. The output should be strictly identical as running the query processor on the command line (see the query processor section above) but that’s a good way to confirm that nothing unexpected happens when PADRE is run by Jetty.

Use the URL /s/padre-sw.cgi:

Validate padre-sw output is the same as using CLI:

Here we’re getting the exact same output as the command line, so we’re good.

Using the XML or JSON endpoint allows bypassing the FreeMarker rendering to control that the Modern UI is building the right data model.

It’s recommended to use JSON because XML has its own quirks (such as requiring XML entities for special characters) which can be misleading.

That’s normal! The technical details are a bit complex, but the Modern UI will re-encode some special characters from the PADRE result packet: \, ", ', <, > and &.

That’s however where our problem lies. Depending on how the value is interpreted in FreeMarker the encoding will or will not show.

If you encounter a problem at this point it can be either a Modern UI bug or a custom hook script that’s doing something wrong. Try disabling all the hook scripts first.

Now that we’ve confirmed that the data model is right we need to look at FreeMarker. FreeMarker will take the data model as is and render it to HTML. Different parameters can affect the rendering (The FreeMarker documentation about charsets might be worth reading)

Your FreeMarker template will probably declare an encoding in the HEAD section, something like <meta charset="utf-8">

This cannot be changed to something else. The Modern UI will always output UTF-8 and there’s no way to change that. If you declare a different encoding in your HTML page, the browser will try to interpret the UTF-8 data as a something else, and it will result in corruption.