Deep Learning, Computer Vision and NLP

Uber used RAG and AI agents to build its in-house Text-to-SQL, saving 140,000 hours annually in query writing time. 📈
Here’s how they built the system end-to-end:

The system is called QueryGPT and is built on top of multiple agents each handling a part of the pipeline.

1. First, the Intent Agent interprets user intent and figures out the domain workspace which is relevant to answer the question (e.g., Mobility, Billing, etc).

2. The Table Agent then selects suitable tables using an LLM, which users can also review and adjust.

3. Next, the Column Prune Agent filters out any unnecessary columns from large tables using RAG. This helps the schema fit within token limits.

4. Finally, QueryGPT uses Few-Shot Prompting with selected SQL samples and schemas to generate the query.

QueryGPT reduced query authoring time from 10 minutes to 3, saving over 140,000 hours annually!

Link to the full article: https://www.uber.com/en-IN/blog/query-gpt/?uclick_id=6cfc9a34-aa3e-4140-9e8e-34e867b80b2b

How Much GPU Memory Needed To Server A LLM ?

This is a common question that consistnetly comes up in interview or during the disscusiion with your business stakeholders.

And it’s not just a random question — it’s a key indicator of how well you understand the deployment and scalability of these powerful models in production.

As a data scientist understanding and estimating the require GPU memory is essential.

LLM's (Large Language Models) size vary from 7 billion parameters to trillions of parameters. One size certainly doesn’t fit all.

Let’s dive into the math that will help you estimate the GPU memory needed for deploying these models effectively.

𝐓𝐡𝐞 𝐟𝐨𝐫𝐦𝐮𝐥𝐚 𝐭𝐨 𝐞𝐬𝐭𝐢𝐦𝐚𝐭𝐞 𝐆𝐏𝐔 𝐦𝐞𝐦𝐨𝐫𝐲 𝐢𝐬

General formula, 𝐦 = ((𝐏 * 𝐬𝐢𝐳𝐞 𝐩𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫)/𝐦𝐞𝐦𝐨𝐫𝐲 𝐝𝐞𝐧𝐬𝐢𝐭𝐲) * 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝 𝐟𝐚𝐜𝐭𝐨𝐫

Where:
- 𝐦 is the GPU memory in Gigabytes.
- 𝐩 is the number of parameters in the model.
- 𝐬𝐢𝐳𝐞 𝐩𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 typically refers to the bytes needed for each model parameter, which is typically 4 bytes for float32 precision.
- 𝐦𝐞𝐦𝐨𝐫𝐲 𝐝𝐞𝐧𝐬𝐢𝐭𝐲 (q) refer to the number of bits typically processed in parallel, such as 32 bits for a typical GPU memory channel.
- 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝 𝐟𝐚𝐜𝐭𝐨𝐫 is often applied (e.g., 1.2) to account for additional memory needed beyond just storing parameters, such as activations, temporary tensors, and any memory fragmentation or padding.

𝐒𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝 𝐅𝐨𝐫𝐦𝐮𝐥𝐚:

M = ((P * 4B)/(32/Q)) * 1.2

With this formula in hand, I hope you'll feel more confident when discussing GPU memory requirements with your business stakeholders.

#LLM

Best article on GenAI getting started
https://blog.bytebytego.com/p/where-to-get-started-with-genai

The matrix calculus for Deep Learning. Very well written. https://explained.ai/matrix-calculus/

Early result of Gemma 2 on the leaderboard, matching Llama-3-70B.

- Full data at leaderboard.lmsys.org
- Chat with Gemma 2 at chat.lmsys.org
- Gemma 2 blog goo.gle/3RLQXUa

Ai / Computer Vision Bootcamp🚀

Learn Ai / Computer Vision from Basics to Deployment from IITian and COEPian.

✅ Build Face Recognition ☺️
✅ Build Ai Object detection 🏍️🚇✈️
✅ Building Social Distancing App
✅ Build Automated Invoice reader 📃
✅Image Classification 🍘🥗
✅ Build Application of Computer Vision in Healthcare, Automotive, retail, Manufacturing and Security, Surveillance. 📸

+40 Hrs sessions.
+12 Weeks.
+13 Tools & Technology.
+7 Projects.
+7 Homework Assignments.
+5 Case studies
+5 Skills.
+5 Domains.

📌 Remote and weekend sessions.
📌 Starting from basics.
📌 Get Certificate.

Duration: 3 months

Attend the 1st FREE session on 11th May: https://chat.whatsapp.com/BibIwuuUEWrGEWdZHYluNe

For registrations: https://aiindia.ai/cv-bootcamp/

Microsoft just casually shared their new Phi-3 LLMs less than a week after Llama 3 release. Based on the benchmarks in technical report (https://arxiv.org/abs/2404.14219), even the smallest Phi-3 model beats Llama 3 8B despite being less than half the size.

Phi-3 has "only" been trained on 5x fewer tokens than Llama 3 (3.3 trillion instead of 15 trillion)

Phi-3-mini less has "only" 3.8 billion parameters, less than half the size of Llama 3 8B.

Despite being small enough to be deployed on a phone (according to report), it matches the performance of the much larger Mixtral 8x7B and GPT-3.5. (Phi-3 mini can be quantized to 4-bits, so it only requires ≈ 1.8GB of memory.)

What is the secret sauce? According to the technical report, it's dataset quality over quantity: "heavily filtered web data and synthetic data".

Next to the 4k context-window version, there's also a phi-3-mini-128K model that supports up to 128k tokens.

Fun fact: Phi-3 uses the same tokenizer with a vocabulary size of 32,064 as Llama 2.

Seems like GPT-4 is back on top with their latest update. Will have to wait and see how it compares in real time tests.

Good material on Developing AI systems in Medical Imaging: https://aiformedicalimaging.blogspot.com/2023/09/things-to-consider-when-developing-ai.html

Retrieval-Augmented Generation for Large Language Models: A survey

This paper is a must read.

It covers everything you need to know about the RAG framework and its limitations. It also lists different state-of-the-art techniques to boost its performance in retrieval, augmentation, and generation.

The ultimate goal behind these techniques is to make this framework ready for scalability and production use, especially for use cases and industries where answer quality matters *a lot*.

These are the key ideas the paper discusses to make your RAG more efficient:

- 🗃️ Enhance the quality of indexed data by removing duplicate/redundant information and adding mechanisms to refresh outdated documents

- 🛠️ Optimize index structure by determining the right chunk size through quantitative evaluation

- 🏷️ Add metadata (e.g. date, chapters, or subsection) to the indexed documents to incorporate filtering functionalities that enhance efficiency and relevance

- ↔️ Align the input query with the documents by indexing the chunks of data by the questions they answer

- 🔍 Mixed retrieval: combine different search techniques like keyword-based and semantic search

- 🔄 ReRank: sort the retrieved documents to maximize diversity and optimize the similarity with a « template answer »

- 🗜️ Prompt compression: remove irrelevant context

- 💡 HyDE: generate a hypothetical answer to the input question and use it (with the query) to improve the search

- ✒️ Query rewrite and expansion to reformulate the user’s intent and remove ambiguity

Link: https://arxiv.org/abs/2312.10997

Think of an LLM that can find entities in a given image, describe the image and answers questions about it, without hallucinating ✨

Kosmos-2 released by Microsoft is a very underrated model that can do that. ☃️ Not only this, but Hugging Face transformers integration makes it super easy to use!
Colab link:
https://colab.research.google.com/drive/1t25qM_lOM-HQG6Wg3aRiF4LOuQMN5lUF?usp=sharing

How big do LLMs need to be able to reason?🤔 Microsoft released Orca 2 this week, a 13B Llama-based LLM trained on complex tasks and reasoning. 🧐 Orca's performance comes from its use of synthetically generated data from bigger LLMs. I took a deeper look at paper and extracted the implementation and other insights.

𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻:
1️⃣ Constructed a new dataset (Orca 2) with ~817K samples using prompts from FLAN, and GPT-4 to generate reasoning responses with the help of detailed system prompts.
2️⃣ Grouped prompts into categories based on similarity to assign tailored system prompt that demonstrate different reasoning techniques.
3️⃣ Replaced the original system prompt with a more generic one, to have the model learn the underlying reasoning strategy (Prompt erasing).
4️⃣ Used progressive learning, starting with finetune Llama on FLAN-v2 (1 ep) , retrain on 5M ChatGPT data from Orca 1 (3 ep), combine 1M GPT-4 data from Orca 1 & 800k new Orca 2 data for final training (4 ep).

𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀:
📊 Imitation learning can improve capabilities with enough data.
🔬 Reasoning and longer generations to get the correct answer help smaller models to compete with bigger LLMs.
💫 Prompt Erasing helped Orca to “learn” reasoning
🎯 Lowest hallucination rates of comparable models on summarization
⚙️ Used packing for training, concatenating multiple examples into one sequence.
👨‍🦯 Masked user & system inputs (prompt) and only used generation for loss
🖥 Trained on 32 A100 for 80h

Paper: https://huggingface.co/papers/2311.11045
Model: https://huggingface.co/microsoft/Orca-2-13b

Its not only about LLMs......

🌟 Microsoft Introduces Florence-2, a Breakthrough in Computer Vision!

👉 Microsoft has just unveiled Florence-2, a revolutionary foundation model designed for various computer vision and vision-language tasks. This new model simplifies the process by using one backbone for multiple tasks. Read more about it in the paper and project details provided below 👇

Key Highlights:

✅ Achieves state-of-the-art performance in various tasks
✅ Employs a unified, prompt-based representation for vision tasks
✅ Features the FLD-5B dataset, boasting over 5 billion annotations with 126 million pictures
✅ Handles detection, captioning, and grounding—all with a single model
✅ Streamlined with a uniform set of parameters governing everything

Finally, we have a hallucination leaderboard! 😍😍

Key Takeaways

📍 Not surprisingly, GPT-4 is the lowest.

📍 Open source LLama 2 70 is pretty competitive!

📍 Google's models are the lowest. Again, this is not surprising given that the #1 reason Bard is not usable is its high hallucination rate.

Really cool that we are beginning to do these evaluations and capture them in leaderboards!

JARVIS-1: Open-Ended Multi-task Agents with Memory-Augmented Multimodal Language Models

abs: arxiv.org/abs/2311.05997
project page: craftjarvis-jarvis1.github.io

"We introduce JARVIS-1, an open-ended agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe."

https://abvijaykumar.medium.com/fine-tuning-llm-parameter-efficient-fine-tuning-peft-lora-qlora-part-1-571a472612c4

https://huggingface.co/blog/4bit-transformers-bitsandbytes

Low-Rank Adaptation of Large Language Models (LoRA) is a training method that accelerates the training of large models while consuming less memory.

🤖 LoRA is like a special trick that helps computers learn better and faster. Imagine a computer trying to learn new things, like recognizing pictures or understanding language. When it learns, it uses something called "weights," which are like little helpers inside the computer.

🚀 Now, LoRA's trick is to make these little helpers work smarter. Instead of changing all the helpers every time the computer learns something new, LoRA only changes a few of them. It's like having a big group of friends, but only a couple of them have to do the hard work, and the others can rest.

💡Here's how it works:
1. The computer has these helpers (weights) that it uses to learn.
2. LoRA makes two special groups of helpers that are smaller and easier to work with.
3. The computer trains these special groups of helpers to learn new things without changing all the original helpers.
4. After training, the computer combines the new helpers with the original ones, like mixing two colors to get a new color.
5. This makes the computer learn faster and doesn't use too much computer memory.

🚀 The good things about LoRA are:
- It helps the computer learn without using too many helpers, so it's faster.
- The original helpers stay the same, so we can use them for different tasks.
- It can work with other tricks that make computers smarter.
- The computer works just as fast when using LoRA, so we don't have to wait.

So, LoRA is like a cool trick that helps computers learn better and faster without making them slow down. It's like having a superhero team of helpers inside the computer! ❤️

Connect with us on WhatsApp: https://whatsapp.com/channel/0029Va8iIT7KbYMOIWdNVu2Q

Dear friends,

I’d like to share a part of the origin story of large language models that isn’t widely known.

A lot of early work in natural language processing (NLP) was funded by U.S. military intelligence agencies that needed machine translation and speech recognition capabilities. Then, as now, such agencies analyzed large volumes of text and recorded speech in various languages. They poured money into research in machine translation and speech recognition over decades, which motivated researchers to give these applications disproportionate attention relative to other uses of NLP.

This explains why many important technical breakthroughs in NLP stem from studying translation — more than you might imagine based on the modest role that translation plays in current applications. For instance, the celebrated transformer paper, “Attention is All You Need” by the Google Brain team, introduced a technique for mapping a sentence in one language to a translation in another. This laid the foundation for large language models (LLMs) like ChatGPT, which map a prompt to a generated response.

Or consider the BLEU score, which is occasionally still used to evaluate LLMs by comparing their outputs to ground-truth examples. It was developed in 2002 to measure how well a machine-generated translation compares to a ground truth, human-created translation.

A key component of LLMs is tokenization, the process of breaking raw input text into sub-word components that become the tokens to be processed. For example, the first part of the previous sentence may be divided into tokens like this:

/A /key /component /of /LL/Ms/ is/ token/ization

The most widely used tokenization algorithm for text today is Byte Pair Encoding (BPE), which gained popularity in NLP after a 2015 paper by Sennrich et al. BPE starts with individual characters as tokens and repeatedly merges tokens that occur together frequently. Eventually, entire words as well as common sub-words become tokens. How did this technique come about? The authors wanted to build a model that could translate words that weren’t represented in the training data. They found that splitting words into sub-words created an input representation that enabled the model, if it had seen “token” and “ization,” to guess the meaning of a word it might not have seen before, such as “tokenization.”

I don’t intend this description of NLP history as advocacy for military-funded research. (I have accepted military funding, too. Some of my early work in deep learning at Stanford University was funded by DARPA, a U.S. defense research agency. This led directly to my starting Google Brain.) War is a horribly ugly business, and I would like there to be much less of it. Still, I find it striking that basic research in one area can lead to broadly beneficial developments in others. In similar ways, research into space travel led to LED lights and solar panels, experiments in particle physics led to magnetic resonance imaging, and studies of bacteria’s defenses against viruses led to the CRISPR gene-editing technology.

So it’s especially exciting to see so much basic research going on in so many different areas of AI. Who knows, a few years hence, what today’s experiments will yield?

Keep learning!
Andrew NG