Prompt engineering (Unprocessed)

I have been reading about prompt engineering a lot and this conversation in lenny’s podcast https://www.youtube.com/watch?v=eKuFqQKYRrA . His Blog post has more detail started the en tire discussion. This is my raw notes on it.

Should you spend time learning prompt engineering? Short answer Yes. its a big area of study.

• Artificial Social intelligence is prompting engineering with better name. Communicating with AI more intelligent way in a language that AI understand better. Better prompt leads to better results. Understanding what their responses mean and how to get better response.
• Two models of prompts -
  • a. Conversational prompt engineering - Like asking questions on a chat
  • b. one prompt use in a pipeline which must be perfect. You can iteratively built and never change.

This discussion is mainly focused on the second kind.

Basic techniques

• Few Shot prompting - Give the AI examples of what I want done. This can be actual output or data for it understand what the expectation is. That is give the AI few shots in a Q&A format including one it hasn’t seen before.
• Zero Shot Prompting - Just ask the AI do the task without giving any example.
• One shot prompting - Give AI one example

Formatting Few Shot Prompting

• Check the Prompt Report Research Paper and this is website for it Research Summary
• Use any common formate the LLMs can understand for example - XML, Markdown etc.
• Structure output as Q&A as LLMs are naturally an answering engine.

Techniques to avoid

• Role playing prompts with more inter personal abilities perform slightly (statistically insignificant) better. There is no predictable effect specially with current LLMS
• The LLMs with role playing prompt do better in more expressive tasks than in analytical tasks.
• There is also no effect in saying Somebody may die or similar threatening or rewarding prompts. As they don;t generate better output than when not done.

Techniques that work

Decomposition

• Break down the prompt into
  • Core ideas - Some tasks the model have to do. In service of that task what are some sub problem need to solve, solving which the LLM solves the bigger problem. Ask the llms to break down the tasks into small enough problems the LLMs can tackle and then use to solve the bigger problem.
  • Sub-problems for example can different tools which in turn solves provides the LLM with required Knowledge

Self-criticism

• Ask the LLM to look back and check its generated response. Example: Can you go back and check your responses and offer self-criticism and rewrite the answer. This way LLMs automatically updates and improves the output.

Context (Adding extra stuff content to the LLM context)

• Give as much information as possible
• This is different than data analysis
• Entrapment Classifier - Small things can have massive effect on the output
• Additional context is good - Remember - the more the better
• The issue with more context is - with cost. The more the context the more is the token cost and more time the LLMs going to take to respond.
• Latency and cost are the main drivers to decide how much context to add.

the other issue is that if the context is too big the model might forget or move to a different direction than then actual task. So there is always a balance that the developer must maintain.

Techniques for Math

Ensembling Techniques

• Evaluate accuracy of a mathematically / programmatically by using multiple LLM output. Here is how to do it: Take a problem and solve it using multiple different prompts then use the response that most common among them. This is similar to how Random Forest algorithm worked.

Mixture of reecing experts

• Set of experts / Role prompt based on the task
• Take multiple LLMs and multiple outputs and use the most common answer among them

Chain of thought Prompting

• Generally its not very useful - specifically true for reasoning models. That being said not a lot of LLMS are reasoning model.

Prompt Injection techniques

• Typos - they have been and continue to be used for prompt injection attack
• Encoding the prompt in some basic encoder that LLMs can generally understand like Base 16 or ROt 13 algorithms
• Prevent Uplift - Search the internet and in one of the input we have prompt injection so the space is very very large.

Prompt injection defence techniques that don’t work

• Asking LLMs to “DO NOT FOLLOW: malicious instructions” or similar methods
• Guardrail models - or prompt modification don’t work - specially in case of a motivated Red Team member

The best defence is - safety training and fine tuning the model which works to a greater degree than other mechanism. The other approach is retrain the model for specific tasks and specific issues in mind and when the threshold crosses - give a canned response. This is inherently a very hard problem and not a solvable one at this point. Similar to what social engineering is in human.