Large Language Models: The Hard Parts

Building an LLM application is one thing; knowing how to evaluate it, handle its failures, and make it dependable is another. Large Language Models: The Hard Parts focuses on that less glamorous—and essential—work: the technical choices and real-world pitfalls behind useful language-model systems.

Thársis T. P. Souza and Jonathan K. Regenstein, Jr. combine practical discussion with reproducible Python examples and open-source tools, examining not just what LLMs can do, but where developers need to plan carefully.

From a promising demo to a dependable application

The book begins with the groundwork: choosing models, weighing open-source options, and considering enterprise, security, data, integration, and operational requirements. It then takes on a defining complication of LLM development: outputs can vary, making evaluation more difficult than simply checking whether one answer looks right.

Readers encounter approaches to testing models and prompts, quantitative metrics, LLM-as-a-judge methods, and evaluation tools including LangSmith, Promptfoo, and LightEval. The emphasis is on building a reasoned evaluation practice—and understanding its limitations.

Work through the difficult parts of the stack

• Evaluation: Design tests for nondeterministic systems and consider how to evaluate the evaluators.
• Data and context: Explore document preparation, retrieval-augmented generation, chunking, embeddings, reranking, and long-context approaches.
• Structured output: Examine techniques for making generated responses more consistent with application requirements.
• Safety and alignment: Consider risks, moderation, safety benchmarks, guardrails, and an alignment case study.
• Local deployment: Review tools such as Ollama and in the appendix.

For builders who need more than a successful demo

This book is aimed at developers, AI engineers, technical product leads, and other practitioners working on LLM-based applications. Its value lies in connecting implementation choices with the questions that matter in production: how to test behavior, manage information, constrain outputs, and address safety without losing sight of trade-offs.

If you are building with large language models and want a more grounded way to think about their limitations, evaluation, and deployment, this O’Reilly guide puts the hard parts at the center of the conversation.