In this post, I go over zero-shot sentiment analysis using Llama 3.2.
First, I wanted to find a good dataset. I went to browse Kaggle datasets and came across <a href="https://www.kaggle.com/datasets/alexandrakim2201/spotify-dataset">this dataset</a>.
The dataset contains user reviews of the Spotify app on Google Play Store.
I love Small Language Models. I can run them on relatively low GPU requirements or even on CPU without worrying too much about how slow it's gonna be, So I wanted to compare both Llama 3.2 1B and 3B on zero-shot classification on this dataset to get a feel of how well they perform and follow instructions. I use the lovely <a href="https://ollama.com/">Ollama</a> API to interact with these models.
The dataset comes in a single CSV file with two columns: <code>Review</code> and <code>label</code>.
Here's the distribution of <code>label</code>
<img src="/zero-shot-llama-3-2/label_dist.png" alt="label_dist">
Here are all the imports we're gonna need:
<div><pre><code class="language-python">import pandas as pd
import ollama
from tqdm.auto import tqdm
from dataclasses import dataclass
from tabulate import tabulate
</code></pre></div>
<div><pre><code class="language-python">df = pd.read_csv("data/DATASET.csv").dropna()
</code></pre></div>
Since I just want to get a feel of how it works and don't really have time to run on all 51K examples in the dataset, I sample I small subset of it.
<div><pre><code class="language-python">SAMPLE_SIZE = 500

df = df.sample(SAMPLE_SIZE, random_state=5578416).reset_index(drop=True)
</code></pre></div>
Here's the distribution of <code>label</code> in the sample
<img src="/zero-shot-llama-3-2/sample_label_dist.png" alt="sample_label_dist">
Then I define <code>num_samples</code> to be the length of my final dataframe
<div><pre><code class="language-python">num_samples = len(df)
</code></pre></div>
I define two prompts to compare between them
<div><pre><code class="language-python">PROMPT_1 = "You are a helpful AI assistant who helps user in syntiment analysis. Given a review in the user message, respond with either `POSITIVE` if the review is positive, or `NEGATIVE` if the review is negative. Respond with ONLY the sentiment and without any elaboration."

PROMPT_2 = "Given the review in the user message, respond with either `POSITIVE` if the review is positive, or `NEGATIVE` if the review is negative. Respond with ONLY the sentiment and without any elaboration."
</code></pre></div>
I define a helper function that returns either <code>POSITIVE</code>, <code>NEGATIVE</code>, or <code>FAILED</code> in case the response wasn't either <code>POSITIVE</code> or <code>NEGATIVE</code>. (This can happen if the SLM doesn't strictly follow our instructions).
<div><pre><code class="language-python">def run(model: str, system_prompt: str, review: str) -> str:
 response = ollama.chat(
 model=model,
 messages=[
 {
 "role": "system",
 "content": system_prompt,
 },
 {
 "role": "user",
 "content": review,
 },
 ],
 options={"temperature": 0},
 )
 answer = response["message"]["content"].strip()
 if answer == "POSITIVE":
 return "POSITIVE"
 if answer == "NEGATIVE":
 return "NEGATIVE"
 return "FAILED"
</code></pre></div>
Then I run a loop going over each model, each prompt, and each sample to run prediction.
<div><pre><code class="language-python">models = ["llama3.2:1b", "llama3.2:3b"]
prompts = {"prompt_1": PROMPT_1, "prompt_2": PROMPT_2}

for model in tqdm(models):
 for prompt_name, prompt in prompts.items():
 df[model + "::" + prompt_name] = ""
 for i in tqdm(range(len(df))):
 df.at[i, model + "::" + prompt_name] = run(model, prompt, df["Review"][i])
</code></pre></div>
I define a class that will hold relevant stats.
<div><pre><code class="language-python">@dataclass
class ModelStats:
 model_name: str
 propmt_name: str
 following_instructions_and_correct_count: int
 following_instructions_and_incorrect_count: int
 not_following_instructions_count: int

 @property
 def following_instructions_and_correct(self):
 return self.following_instructions_and_correct_count / num_samples

 @property
 def following_instructions_and_incorrect(self):
 return self.following_instructions_and_incorrect_count / num_samples

 @property
 def not_following_instructions(self):
 return self.not_following_instructions_count / num_samples
</code></pre></div>
Then I compute these stats:
<div><pre><code class="language-python">model_stats = []
for model in models:
 for prompt_name in prompts.keys():
 col = model + "::" + prompt_name
 correct_count = (df[col] == df["label"]).sum()
 incorrect_count = (df[col] != df["label"]).sum()
 not_following_instructions_count = (df[col] == "FAILED").sum()
 model_stats.append(
 ModelStats(
 model_name=model,
 propmt_name=prompt_name,
 following_instructions_and_correct_count=correct_count,
 following_instructions_and_incorrect_count=incorrect_count,
 not_following_instructions_count=not_following_instructions_count,
 )
 )
</code></pre></div>
Now, I can print all the collected results in a nice format:
<div><pre><code class="language-python">print(
 tabulate(
 [
 ["Model", *[s.model_name for s in model_stats]],
 ["Prompt", *[s.propmt_name for s in model_stats]],
 [
 "Following Instruction &#x26; Correct",
 *[f"{s.following_instructions_and_correct:2.2%}" for s in model_stats],
 ],
 [
 "Following Instruction &#x26; Incorrect",
 *[
 f"{s.following_instructions_and_incorrect:2.2%}"
 for s in model_stats
 ],
 ],
 [
 "Not Following Instructions",
 *[f"{s.not_following_instructions:2.2%}" for s in model_stats],
 ],
 [
 "Correct (Count)",
 *[s.following_instructions_and_correct_count for s in model_stats],
 ],
 [
 "Incorrect (Count)",
 *[s.following_instructions_and_incorrect_count for s in model_stats],
 ],
 [
 "Not Following Instructions (Count)",
 *[s.not_following_instructions_count for s in model_stats],
 ],
 ],
 tablefmt="fancy_grid",
 )
)
</code></pre></div>
And this produces:
<img src="/zero-shot-llama-3-2/llama-3-2.png" alt="image">
We can see <code>prompt_2</code> is producing slightly better results that <code>prompt_1</code>.
We can also see that the 3b model is consistently outperforming its 1b sibling, and both of them don't follow the instruction very rarely.
Here's some anallysis on some correct vs incorrect predictions on different sequemce lengths
<img src="/zero-shot-llama-3-2/llama1b-mistakes-seqlen.png" alt="llama1b-mistakes-seqlen">
<img src="/zero-shot-llama-3-2/llama3b-mistakes-seqlen.png" alt="llama3b-mistakes-seqlen">
Here's a link to the <a href="https://github.com/KhaledEssam/spotify-user-reviews">GitHub repo</a> that has the code.

Zero-shot Sentiment Analysis using Llama 3.2

UnionFind is one of my favorite algorithms. Here, I go into the code for implementing it.
<div><pre><code class="language-python">class UF:
 def __init__(self, size: int):
 self.size = size
 self.parent = [i for i in range(size)]
 self.component_sizes = [1 for _ in range(size)]
 self.num_components = size

 def find(self, p: int) -> int:
 """
 Find the root of the node and do path compression along the way.
 :param p: node which you want to find the root of
 :return: root node of the component to which `p` belongs
 """
 root = p
 while root != self.parent[root]:
 root = self.parent[root]
 current = p
 while current != root:
 nxt = self.parent[current]
 self.parent[current] = root
 current = nxt
 return root

 def connected(self, p: int, q: int) -> bool:
 return self.find(p) == self.find(q)

 def component_size(self, p: int) -> int:
 return self.component_sizes[self.find(p)]

 def unify(self, p: int, q: int):
 if self.connected(p, q):
 return
 r1 = self.find(p)
 r2 = self.find(q)

 smaller, larger = sorted([r1, r2], key=lambda x: self.component_sizes[x])
 self.component_sizes[larger] += self.component_sizes[smaller]
 self.parent[smaller] = larger
 self.component_sizes[smaller] = 0

 self.num_components -= 1
</code></pre></div>

UnionFind

Natural language processing (NLP) has been rapidly evolving over the past decade. From chatbots to virtual assistants, NLP is transforming how machines understand and respond to human language. In this post, we explore the future of NLP and the cutting-edge technologies that are shaping this field.
As AI-driven communication becomes more sophisticated, researchers are focusing on improving language understanding, context awareness, and sentiment analysis. Large language models like GPT-4 have already demonstrated the potential for conversational AI, but what’s next? Will machines ever fully comprehend the nuances of human speech? Let's dive into the key trends and future directions in NLP.

My Blog

Zero-shot Sentiment Analysis using Llama 3.2

UnionFind

The Future of Natural Language Processing