Turning Existing Content into New Category Pages

At Thought Catalog, one of our sites was Quote Catalog, a library of quotes our editors had been hand-curating for years. Every quote had been chosen by a person, attributed and tagged. It was some of the best content we had, and most of it was hard to find. Someone searching for quotes about moving on, or deep love quotes, landed on someone else’s page, even when we had dozens of better quotes on exactly that subject sitting in our database.

The business wanted a bigger search footprint, and the obvious way to get one is to write more. We didn’t need more content. We needed more doors into the content we already had. A category page, a page that groups the quotes on one subject, is a door, and the question was how to build a lot of good ones without an editor assembling each page by hand and then maintaining it forever.

The answer was Elasticsearch’s percolator, which turns search around. Normally you store documents and run queries against them. The percolator stores queries and runs documents against them. For us, every category page became a stored query, and every new quote found its own way onto every page it belonged on.

Reading the Archive

Before we could decide what pages to build, we had to understand what we had. The quotes lived in MySQL, so we worked from a dump of the database rather than the live site.

Each quote carried three to five tags, things like the topic, the mood or the person who said it, added by editors over years. That’s a lot of signal, and also a lot of noise. Years of tagging by different people produce the same idea under several names, tags that are too broad to mean much and tags used exactly once. We ran the quotes and their tags through spaCy, a basic NLP library, to see which tags really described the same thing and what the quotes themselves were about. Then we queried the dump, stored the IDs of the quotes that matched each candidate topic, and looked at those sets directly. How many quotes did each candidate have? How much did two candidates overlap? Was a topic deep enough to carry a page, or would it be twelve quotes and a lot of white space?

That step mattered more than anything we did later. A category page is only worth building if the archive can fill it with good quotes, and the only way to know is to count.

Finding the Demand

Knowing what we had was half the question. The other half was what people were searching for. We used Ahrefs to research the opportunities, the phrases people actually typed into Google that a quote page could answer.

Put the two lists side by side and the work picks itself. A phrase with real search volume and a deep set of quotes behind it was a page worth making. A phrase with volume and nothing behind it was a page we couldn’t fill, and a deep topic nobody searched for was a page nobody would find. The best opportunities were in the overlap, where demand and supply met and nobody had built the door yet.

A Category Is a Query

Once we knew which pages to build, we didn’t build them as lists. We curated a query for each category and stored the query in the database. A category page was whatever that query returned, rendered as quote cards.

That’s a small decision with a big consequence. A list is a snapshot, and it starts going stale the day an editor finishes it. A query is a definition. It says what belongs on the page rather than what happened to be on it when someone last looked, and an editor can read it, adjust it and see the page change.

Here’s roughly what that looks like in Elasticsearch. The example is illustrative, not our production mapping. The index of categories has a field of type percolator, which, in the docs’ words, “understands the query dsl and stores the query in such a way that it can be used later on” (Elastic):

PUT /categories
{
  "mappings": {
    "properties": {
      "query": { "type": "percolator" },
      "slug":  { "type": "keyword" },
      "text":  { "type": "text" },
      "tags":  { "type": "keyword" }
    }
  }
}

The text and tags fields are there because a stored query has to know the shape of the documents it will be matched against. Then each category is one document, holding its query:

PUT /categories/_doc/moving-on
{
  "slug": "moving-on-quotes",
  "query": {
    "bool": {
      "should": [
        { "terms": { "tags": ["moving on", "letting go", "breakups"] } },
        { "match_phrase": { "text": "move on" } }
      ],
      "minimum_should_match": 1
    }
  }
}

The query can use the editors’ tags, the words in the quote, or both, which is how a category can reach quotes that were never tagged for it but plainly belong.

Letting New Quotes Find Their Pages

The part that made this worth doing was what happened next. When an editor added a quote, we didn’t ask which pages it should go on. We asked Elasticsearch, by percolating the quote against every stored category:

GET /categories/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "text": "You can't start the next chapter of your life if you keep re-reading the last one.",
        "tags": ["moving on", "life"]
      }
    }
  }
}

The response is the list of categories whose queries match. If a quote fit a category, it was already on that page. Nobody had to remember that the new quote about letting go also belonged on the moving-on page, the breakup page and the self-love page. The archive kept filing itself, and every page we’d built kept getting better without anyone touching it.

Elasticsearch originally had a separate percolate API, and in version 5.0, in 2016, the team replaced it with the percolate query, because, as their release post put it, “percolation is just a special form of search” (Elastic). A percolation became an ordinary search request, with the same scoring, highlighting and pagination as any other, which is what makes it easy to drop into an existing search stack.

It isn’t free. Elastic’s own docs warn that “percolate queries do not scale in the same way as other queries,” and part of what keeps them fast is that Elasticsearch extracts terms from each stored query at index time, so it can skip queries that couldn’t possibly match before running the rest. Some queries, like wildcards, can’t be extracted and lose that shortcut. With a curated set of category queries, each written by a person, that was never a problem for us. With hundreds of thousands of loose ones, it would be.

What It Did

By my memory, traffic to the site rose somewhere between 15 and 20 percent from the new category pages. The bigger change was in how the team worked. Building a page went from an editor assembling and maintaining a list to an editor curating one query, and the archive did the rest.

Takeaways

If I were doing this again, I’d start the same way, by reading the archive before deciding what to build. The analysis of what we already had, and how deep each topic really was, decided which pages were worth making far more than the search data did. I’d keep categories as stored queries, because a query is a definition someone can read and own, and a list is a snapshot someone has to maintain. And I’d use the percolator for the same reason we did. The expensive part of a category page isn’t building it. It’s keeping it full, and the percolator does that for free every time a new quote comes in.

We didn’t need more content. We needed more doors into the content we already had.

Sources

1. Elastic, “Percolate query,” Elasticsearch documentation
2. Elastic, “Elasticsearch 5.0.0-alpha2 released,” May 2016