LLM prompt techniques from Financial Times

I stumbled upon a talk by a principal engineer at Financial Times, Katie Koschland. She discussed how their authors apply large language models that her team is developing, the challenges they face, and shared some advice on prompt engineering.

Photo by Min An: https://www.pexels.com/photo/close-up-photo-of-vintage-typewriter-1425146/

Many people are wary of AI in journalism due to hallucinations and a lack of credibility, especially given that many automatically generated texts are distributed without disclaimers indicating they were created by a machine.

But that is only one side of the coin. Katie argued that LLMs can be tools like code linters, code generators, and other instruments programmers have used for decades. Similarly, authors can use AI to analyze, search, and filter vast amounts of historical data to speed up their work.

I don't promise any revelations for people experienced with LLMs. However, I thought it'd be nice to have a small cheat sheet, which encouraged me to write this post.

There it goes:

1. Prime the LLM to take a role: "You are a very experienced journalist at the Financial Times focusing on British politics."
2. Allow the LLM to fail: "Please answer the following question to the best of your ability. If you are unsure or don't have enough information to provide a confident answer, simply say 'I don't know' or 'I'm not sure'."
3. Provide the LLM with a few crafted examples to guide the LLM's responses. Katie suggests wrapping examples in the XML tags to help the LLM differentiate between the examples and the rest of the prompt: "Here are some good examples: <example1>...</example1><example2>...</example2><example3>...</example3>"
4. Allow the LLM to think through the problem step-by-step: "Before answering the question, please think about it step-by-step within <thinking></thinking> tags. Then, provide your final answer within <answer></answer> tags."

Regarding the last technique, I think that so-called reasoning models are built on this idea. It looks very similar to what they are doing.

Here's the final prompt:

You are a very experienced journalist at the Financial Times focusing on British politics. (01)

<initial prompt>

Please answer the following question to the best of your ability. If you are unsure or don’t have enough information to provide a confident answer, simply say “I don’t know” or “I’m not sure”. (02)

Here are some good examples:

<example>Example 1</example>
<example>Example 2</example>
<example>Example 3</example> (03)

Before answering the question, please think about it step-by-step within <thinking></thinking> tags. Then, provide your final answer within <answer></answer> tags. (04)