Multi Agent Frameworks (LLM)

atomic-agents (OSS, simple, based on instructor

- instructor and schema names = Bad Schemas could break your LLM Structured Outputs - Instructor

magnetic-one (Microsoft)

TinyTroupe  (Microsoft)

TinyTroupe is an experimental Python library designed to simulate individuals, referred to as TinyPersons, with distinct personalities, interests, and goals.

amazon - https://github.com/awslabs/multi-agent-orchestrator

    - https://aws.amazon.com/bedrock/agents/

pydantic fw - https://ai.pydantic.dev/

(related) Gemini - has Plans and Approvals? = https://developers.googleblog.com/en/the-next-chapter-of-the-gemini-era-for-developers/

Scaling Functionality (large number of FCs) = github.com/microsoft/semantic-kernel/discussions/9947
- use a Vector store
- semantic search to find the related function calls, so only send the related ones to the LLM

Semantic Kernel - Agents = https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/?pivots=programming-language-csharp#what-is-an-ai-agent

AI Engineering[Book] [recommended by Zeki]

Anthropic - Building Effective Agents = https://www.anthropic.com/research/building-effective-agents

Agent paper by Google = 1735887787265

LLMs: Lessons Learned and Evaluation

 Applied LLMs: What we've learned from a year of building with LLMs
https://applied-llms.org/ 

Lessons Learned Building Products Powered by Generative AI

https://tech.buzzfeed.com/lessons-learned-building-products-powered-by-generative-ai-7f6c23bff376

EvalGen - Who validates the validators - https://arxiv.org/abs/2404.12272

Your AI Product Needs Evals

https://hamel.dev/blog/posts/evals/

Asian classic texts

Journey to the West

 https://en.m.wikipedia.org/wiki/Journey_to_the_West

AI - Reinforcement Learning and Robotics

AI - Reinforcement Learning and Robotics

# Robotics

- Majko Physics Engine - https://github.com/google-deepmind/mujoco

- book: Sutton & Barto - "Reinforcement Learning: An Introduction"

- free eBook

- Robot Operating System (ROS) = https://github.com/ros-infrastructure/www.ros.org

War-time and Spy Books

Exile's Return

cowley - killing thatcher (doug)

Seventeen Moments of Spring (soviet spy)

(done) operation mince meat

Food and Cooking - recipes

from Kitchen Confidential- Anthony Bourdain

Orwell's Down and Out in Paris and London. 

Nicolas Freeling's The Kitchen, 

David Blum's Flash in the Pan, 

the Batterberrys' fine account of American restaurant history, On the Town in New York, 

Joseph Mitchell's Up in the Old Hotel.

Read the old masters: Escoffier, Bocuse et al.

as well as the Young Turks: Keller, Marco-Pierre White

———————

Jamie Oliver recipes

Caponata - aubergine stew

https://www.jamieoliver.com/recipes/vegetables-recipes/incredible-sicilian-aubergine-stew-caponata/

graphics programming

taichi

 - language for computer graphics research

https://docs.taichi-lang.org/docs/overview

geography

Influences of Geographic Environment by Ellen Churchill Semple

https://www.gutenberg.org/ebooks/15293

AI: algorithms

K-Means clustering [can be expensive]

- elbow method - calc how many clusters, but can be expsensive

alt: DBSCAN which calculates how many clusters to use (density-based clustering non-parametric algorithm)

https://en.wikipedia.org/wiki/DBSCAN

books read by Jordan Peterson

the parasitic mind

return of the god hypothesis

the myth of mental illness

factfulness

east of eden

the madness of crowds

the rational optimist

an anthropologist on mars

AI: LLMs, platforms (hardware) and projects

(AI in Business) - Applied Artificial Intelligence

https://www.bookdepository.com/Applied-Artificial-Intelligence-Where-AI-Can-Be-Used-Business-Francesco-Corea/9783319772516?ref=grid-view&qid=1592465308969&sr=1-1 

Web element detection

OpenDILabCommunity/webpage_element_detection · Hugging Face

Layout parser - doc ingester

https://layout-parser.github.io/

YOLO Object Detection
https://www.v7labs.com/blog/yolo-object-detection

Data generation - Langchain

Synthetic data generation | 🦜️🔗 Langchain

DistilBERT: Faster version of BERT

[1910.01108] DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (arxiv.org)

DBRX - a 'mixture of experts' LLM

Introducing DBRX: A New State-of-the-Art Open LLM | Databricks

Hermes-2 Mistral 7B - good for "Function Calling" (so LLM delegates creating/modifying an app, to the client)

https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B

multimodal LLMs

HuggingFaceM4/idefics2-8b-chatty · Hugging Face

HuggingFaceM4/idefics2-8b · Hugging Face

HPT 1.5 Air: A New Open-Sourced 8B Multimodal LLM with Llama 3

Datasets - Roboflow (like HuggingFace for data)

https://public.roboflow.com/

Accelerating Hugging Face Transformers with AWS Inferentia2

AWS Inferentia2 (inf2 AWS instances - cheaper than g5)

Prompt hacking and testing

HackAPrompt

https://github.com/Giskard-AI/giskard

Token counts - for cost estimates

https://huggingface.co/Xenova/claude-tokenizer

https://huggngface.co/spaces/Xenova/the-tokenizer-playground

https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-prepare.html

LLM powered autonomous agents

https://lilianweng.github.io/posts/2023-06-23-agent/

Building a NN from scratch

MNIST Neural Network From Scratch (kaggle.com)

Data science and data trends (via ML) Python libraries

https://www.marktechpost.com/2024/05/15/10-python-packages-revolutionizing-data-science-workflow/

evaluating LLMs

prometheus-2

https://github.com/prometheus-eval/prometheus-eval

LLM GitHub accelerator projects 

https://github.blog/2024-05-23-2024-github-accelerator-meet-the-11-projects-shaping-open-source-ai/

distilbert - can train it to classify intent or toxic

https://huggingface.co/distilbert/distilbert-base-uncased

Prompt engineering guides

- The Prompt Report - https://arxiv.org/pdf/2406.06608

- https://www.promptingguide.ai

- Anthropic - [Answer Key] Anthropic's Prompt Engineering Interactive Tutorial [PUBLIC ACCESS] - Google Spreadsheets

- Claude Prompt Engineering & Function Calls

- Claude - Tool Use

Tiny Multi-modal LLM - regions + OCR

microsoft/Florence-2-large · Hugging Face

AWS Bedrock Tips

- batch inference

https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html

AWS Inferentia - speculative decoding

Faster LLMs with speculative decoding and AWS Inferentia2 | AWS Machine Learning Blog

AWS Prompt Routing and prompt caching

- can auto pick an LLM?

Reduce costs and latency with Amazon Bedrock Intelligent Prompt Routing and prompt caching (preview) | AWS News Blog

AI Papers, Books and Datasets

Learning Transferable Visual Models From Natural Language Supervision = 
https://arxiv.org/abs/2103.00020

Free Datasets

https://www.v7labs.com/blog/best-free-datasets-for-machine-learning

UI Modelling - annotated UIs

https://uimodeling.github.io/

Datasets - Roboflow (like HuggingFace for data)

https://public.roboflow.com/

MoE (Mixture of Experts)

LLM Mixture of Experts Explained (tensorops.ai)

An Introduction to Vision-Language Modeling

Vision AI - VLMs and CNNs
- Vision Language Models
- Convolutional Neural Networks

https://arxiv.org/pdf/2405.17247

Multi-modal LLMs

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (cambrian-mllm.github.io)

Movies: rated [for collaborative filtering]

MovieLens Latest Datasets | GroupLens

The Society of Mind

Book by Marvin Minsky

AI Engineering[Book] [O'Reilly] [Zeki recommends]

same author: Designing Machine Learning Systems- DMLS focuses on building applications on top of traditional ML models, which involves more tabular data annotations, feature engineering, and model training

gen-ai news and updates

 

LocalLLaMA - gen-ai on reddit

https://www.reddit.com/r/LocalLLaMA/

LLM Leaderboard - https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Superintelligence

the beginning of infinity

The Business Case for AI: A Leader's Guide to AI Strategies, Best Practices & Real-World Applications

Book by Kavita Ganesan

- but neglects challenge of culture change and ethics

- profit vs ethics

.