Skip to content
ilian h. herzi

ilian h. herzi

Contact me

bitwise.tarzan

LangChain Llama Jokester

Discussing & using the principles of LangChain to build a comedic conversational bot around Llama 2.

· Originally published on Substack

TL;DR: LangChain is a framework for attaching different capabilities to LLMs that allow these LLMs to be aware of specific datastreams and interact with their environment. In this way, LangChain helps developers make powerful LLM based applications.

LangChain: Chains

In LangChain, everything is a “Chain” or a modular component with a nice API. I think the a chain is best explained with a diagram for a conversational chat bot.

Example of how an LLM can be combined with a retrieval and memory chain component. [1]. (Ultimately I’m hoping that the tracing in LangSmith becomes publicly available so it’s easier to see the tree chain we can build but for now I’m constrained to the diagram from the LangChain docs.)
Example of how an LLM can be combined with a retrieval and memory chain component. [1]. (Ultimately I’m hoping that the tracing in LangSmith becomes publicly available so it’s easier to see the tree chain we can build but for now I’m constrained to the diagram from the LangChain docs.)

In the above example for a conversational chat bot, we have the idea of a making a “Memory” chain and and a “Retrieval” chain which may be used depending on the question (perhaps we want to search our documents of our journal for instance).

Notice how this chain is then connected to the “LLM” Model chain which takes as input a prompt combined from the Memory & Retrieval chain and outputs an answer. As long as we follow the APIs associated with each Chain the idea is that we can keep combining chains for whatever our specific use case is!

Jokester bot

Personally, I like working through an example, so let’s go ahead and build a chain and run it. Our goal is to build a “Jokester” bot, where we’ll try to ask for serious answers but we’ll keep getting ridiculous responses back. This is based on the the LangChain demo repo I made, feel free to play around with the code.

We’ll be using a “Model” chain, a “Memory” chain, and a “Prompt” chain to get us started. See diagram 1 for how the pieces will connect.

Diagram 1: Example of linking multiple chains. The question will be given to our LLM model along with the prompt template and memory context chains.
Diagram 1: Example of linking multiple chains. The question will be given to our LLM model along with the prompt template and memory context chains.
  • A “Model” chain: refers to models that take in some input text and can output some input text. For this tutorial, it’s just our LLM.
  • A “Prompt” chain: refers to the inputs of the model chain. They instruct how the input from the user will look and can be defined in templates for what the input should have.
  • A “Memory” chain: An abstraction to maintain a memory state of the conversation. There is short term memory and long term memory. This example deals with short term (other methods for long term require vector stores which is a component for another tutorial). A good way to think about conversational memory is like so (borrowed from pinecone.io’s learn series for conversation memory)
Pinecone’s description of using a conversatinal memory chain. [2]
Pinecone’s description of using a conversatinal memory chain. [2]

In code the high level way to implement this is in the run(query)method in demo.py

def run(query: str) -> None:
    # Prompt chain
    prompt = build_prompt_template()
    # Llama model using a hugging face pipeline
    pipeline = build_llama_hugging_face_pipeline(
        model_id=LLAMA_MODEL_ID,
        device=f"cuda:{cuda.current_device()}" if cuda.is_available() else "cpu",
    )
    llama_llm = HuggingFacePipeline(
        pipeline=pipeline,
    )
    # Memory chain
    memory = ConversationBufferWindowMemory(k=MEMORY_WINDOW)
    # LLM chain
    llm_chain = LLMChain(
        llm=llama_llm,
        prompt=prompt,
        verbose=False,
        memory=memory,
    )
    output = llm_chain.predict(human=query)
    print(output)
    print(memory.buffer)

Finally, let’s run our model and see what happens! I’m running this in Google colab but feel free to run the notebook wherever you’d like.

This is a bit interesting (and strange) because I never asked the model “Why do huans have to work”. I'm guessing the model is overindexing on the example I provided in the template.
This is a bit interesting (and strange) because I never asked the model “Why do huans have to work”. I'm guessing the model is overindexing on the example I provided in the template.
And we see that our jokester bot remembers that Elon walked on the moon!
And we see that our jokester bot remembers that Elon walked on the moon!

And if we rerun the cells and ask who walked on the moon:

Note that rerunning the cells on a fresh seed with a high temperature will result in significantly different responses however with short-term memory and prompt engineering the style feels the same.

In this post we went through an example of building a conversational jokester with LangChain and Llama 2. We discussed some of the principles of LangChain and showed off our “not-so-funny” jokester. Hope you enjoyed it!

Note this post was heavily inspired by [3]. Thanks @murtuza753!

References:

[1] LangChain documents

[2] Pinecone learn series on conversational memory

[3] Llama 2.0 with LangChain series by @murtuza753