AI Engineering with LLMs: Advanced

Level 1, path 1 (part 2 of 2) of the BitByte AI path, for learners who finished the beginner course or have built a provider-neutral LLM feature with local retrieval, tool calling and regression tests. A demo that answers well is not a system you can release. This course teaches how to measure retrieval, control agents, test judges, secure and observe the application, deploy it with a rollback plan, and write down an honest release decision.

You get a written tutorial in English and Bangla, hand-picked videos, and three exams: two quizzes and a harder final exam of code reading, output prediction, debugging and design choices. The tutorial includes a graded practice lab and the KnowledgeOps capstone. Everything runs offline with fake models, toy embeddings and mocked services; no paid API, GPU or internet is needed, fine-tuning is covered as concepts and a dry run with no real GPU training, and Docker files are shown but not run. The course is educational: facts were checked on 12 October 2026, model names, prices and SDK details change, and the Bangla text was written by AI. It makes no promise about jobs.

• Compare dense, keyword and hybrid retrieval on a labelled corpus, add reranking, and recommend a method from ranking metrics and error analysis
• Build a local MCP server and client with validated inputs and bounded results, and review MCP security and interoperability risks
• Control an agent with step limits, explicit state, tool allowlists and human approval, and stop runaway execution
• Design evaluation sets with holdouts and release gates, and calibrate an LLM judge or mark it experimental, checking for length and position bias
• Decide when fine-tuning is justified, explain LoRA and PEFT with a toy, and plan a dry-run experiment while checking leakage and model rights
• Add structured logs, traces, usage records and monitoring with redaction, budgets and incident handling
• Threat-model the application, test prompt injection and exfiltration, enforce permission-aware retrieval and cross-tenant isolation, and plan ingestion and data freshness
• Deploy locally with health checks, streaming, scaling limits and rollback, then assemble release evidence and defend a go, conditional-go or no-go decision

• The beginner course AI Engineering with LLMs: Beginner, or equivalent experience building a provider-neutral LLM feature with local retrieval, tool calling and regression tests
• Python 3.12 and a terminal, with comfort in writing and running tests
• No API key, GPU or paid account: everything runs offline with fake models and toy embeddings