Mistral AI Releases OpenSource MoE Model

Today’s top AI Highlights:

1. Mistral AI releases an 8x7B MoE model as an 87 GB torrent.

2. Deepgram has released the fastest and most accurate speech-to-text API

3. NotebookLM, Google’s research assistant, is now powered by Gemini Pro

4. OpenAI plans to expand its business in India to tap into the next billion AI users

5. AnythingLLM: A private ChatGPT for chatting with your organization’s data

& so much more!

Read time: 3 mins

Latest Developments 🌍

Mistral’s OpenSource MoE with GPT-4 like similarities 🕸️

In a move that has intrigued the AI community, Mistral AI has open-sourced a Mixture of Experts (MoE) model, presenting an alternative yet efficient approach to LLMs. This release, featuring a total of 8 experts with 7 billion parameters each, offers a unique perspective on model scaling and specialization in AI.

Key Highlights:

1. The MoE model by Mistral AI consists of 8 experts, each with 7 billion parameters. This model is designed to split complex tasks into smaller segments, handled by these specialized mini-models or 'experts'. This structure is aimed at improving efficiency and accuracy in LLMs.

2. The model operates with impressive specs, including a dimension of 4096, 32 layers, and a hidden dimension of 14336. The model's gating network plays a crucial role in routing input to the appropriate experts, based on a compatibility scoring system.

3. Mistral's model shares architectural similarities with GPT-4 but is notably scaled down. It features 42 billion total parameters compared to GPT-4's 1.8 trillion, 32k context, and each expert in the MoE model has 7 billion parameters, a considerable reduction from GPT-4's 166 billion per expert.

4. Compared to SOTA models like Yi-34B and Llama 2 70B, it performs exceptionally well and even outperforms these models in certain benchmarks, showcasing robust capabilities in various language understanding and reasoning tasks.

Fastest and Most Accurate STT API 📝

Deepgram has released its latest and most sophisticated speech-to-text API, Nova-2, the fastest and most accurate yet. It outperforms all other APIs including OpenAI Whisper (large) in terms of speed and accuracy while being significantly cost-effective. Nova-2 is now available in English (both pre-recorded and streaming audio) for early access customers.

Key Highlights:

1. Nova-2 boasts 18% more accuracy than OpenAI Whisper (large), and a 36% relative improvement in Word Error Rate (WER). For pre-recorded audio, the model's inference time is remarkably fast, ranging from 5 to 40x faster than other models. Additionally, it maintains a consistent WER below 10% across diverse audio domains.

2. Nova-2 remains highly affordable, priced at $0.0043 per minute for pre-recorded audio, which is 3 to 5x more cost-effective than other options. The model's training is extensive, utilizing nearly 6 million diverse resources and covering over 100 domains with 47 billion tokens, making it one of the most deeply trained Automatic Speech Recognition models available.

3. The API features significant advancements such as improved speaker diarization, smart formatting, and support for filler words. It also includes domain-specific language models for better summarization.

Gemini Now Powers NotebookLM 🏋️‍♀️

NotebookLM, Google’s AI research assistant to make any document a host of information, is now available in the US and is powered by Gemini Pro for better document understanding and reasoning. It has been supercharged with a suite of new features to enhance your productivity and creative processes.

Key Highlights:

1. NotebookLM features a new noteboard space where you can pin quotes, excerpts, or your written notes for easy reference. This space enhances the ability to save and organize important information and ideas.

2. The tool dynamically suggests actions based on your activity, such as summarizing text or helping understand complex ideas. This feature aims to streamline the process of reading, note-taking, and writing by providing context-sensitive assistance.

3. NotebookLM offers new tools for organizing curated notes into structured documents. You can create outlines, study guides, or other formats by selecting notes and providing instructions to the tool. This feature facilitates the transformation of notes into coherent and organized documents.

OpenAI Plans to Expand to India 🪐

OpenAI has engaged Rishi Jaitly, the former head of Twitter India, as a senior advisor to aid in discussions with the Indian government regarding AI policy, indicating OpenAI's increasing focus on establishing a presence in India, a key market given its status as the world's second-largest internet market with over 880 million users. OpenAI’s CEO, Sam Altman, met with Indian Prime Minister Narendra Modi earlier this year, highlighting the company's interest in the country.

Rishi Jaitly's role, while not formally confirmed as an OpenAI employee, involves advising the company on navigating the Indian policy and regulatory landscape. The broader context of OpenAI's move into India comes at a time when the country is perceived as lagging in AI development, with its AI startups having raised around $4 billion which is significantly less than other major markets like China. Despite this, there's a growing recognition of the potential for AI innovation in India.

Tools of the Trade ⚒️

• AnythingLLM: A full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as reference during chatting. Choose which LLM or Vector Database you want to use as well as supporting multi-user management and permissions.

• Vision on Julius AI: Julius, the AI-powered data analyst and math helper can now see images. Upload images to solve math problems, turn screenshots into data, create user interfaces, and much more.

• Flourish Studio: A platform for creating engaging data visualizations, transforming data into interactive charts, maps, embeds, presentations, and more. It is designed for ease of use with customizable templates.

• Lean Copilot: AI Copilot to assist in proving mathematical theorems, offering features like tactic suggestion, proof search, and premise selection. It is designed for advanced proof automation in Lean, supporting customization and various models for theorem-proving tasks.

Hot Takes 🔥

1. Extreme 1: “Deepmind faked the evals and demo. Gemini sucks” Extreme 2: “OpenAI is done. Google is back. Bard will run Gemini for free and burn down chatGPT because of margins on compute chip” Reality: Gemini is cool. The first model that genuinely is comparable to GPT 4. Real accomplishment. Especially that it was just a dense model. Marketing was overboard, but Deepmind is known for aggressive PR. Demos like the multimodal video in reality will be possible in less than a year. ~ Aravind Srinivas

2. One of the most interesting (and also frightening) things about AI is how difficult it is to predict what will happen. My kids ask me what's going to happen, and all I can say is that there will be huge changes, and I can't predict what they'll be. ~ Paul Graham

Meme of the Day 🤡

That’s all for today!

See you tomorrow with more such AI-filled content. Don’t forget to subscribe and give your feedback below 👇

Real-time AI Updates 🚨

⚡️ Follow me on Twitter @Saboo_Shubham for lightning-fast AI updates and never miss what’s trending!!

PS: I curate this AI newsletter every day for FREE, your support is what keeps me going. If you find value in what you read, share it with your friends by clicking the share button below!