Building a Local LLM · 18/30 · API and Document Search
Full series and learning sequence · Practice code ZIP
Keep a fixed set of questions so a model replacement or update can be compared against the same expectations. Include previous successes, previous failures and questions with no answer in the documents. Start with a small evaluation sheet that separates retrieval quality from answer quality.
Prepare the search and answer structure for parts 16 and 17. Pick tasks that you assign frequently and use the same questions and materials for all candidates.
Create 20 assessment questions
| bundle | Number of questions | Grading criteria |
|---|---|---|
| Summary of original text | 4 | Preservation of numbers, times, and proper nouns |
| information extraction | 4 | Required fields and null handling |
| Document Search | 4 | Correct answer chunks included in top candidates |
| unanswerable question | 4 | Do not create facts or phone numbers that do not exist |
| Simple code modification | 4 | Actually passed the designated test |
Divide the frequently encountered questions during development and the questions to be used for final comparison. If you evaluate the sentences used in learning as is, you can see high actual performance. The same data and scoring criteria are applied to the models being compared.
Grading criteria and execution conditions
The storage period is recorded as 1 point if 14 days is correct, and 0 points if any other number is guessed. You will receive 1 point if you answer that you cannot verify a phone number that is not in the document. Set aside the writing score. Results that are incorrect on important facts will not be passed as a speech score.
id,task,expected,model,context,passed,latency_seconds,note
backup-01,보관 기간,14일,,,,,
backup-02,담당자 전화번호,문서에 없음,,,,,
extract-01,담당자 추출,null,,,,,
search-01,백업 주기 검색,backup.txt#1,,,,,CSV in ZIP is a recording format that does not contain measurement results. Write down the model tag, ID, quantization, num_ctx, prompt version, and runtime version. You should leave comparison conditions so you can later look into why the numbers are different.
Separate evaluation of search, creation, and speed
Recall@k is a search indicator that checks whether the correct answer is in the top k. If you find the evidence and the answer is wrong, look at the creation steps and instructions. If evidence is missing, first review a combination of chunks, embedding, and keyword search.
Requests for initial loading and repeated requests after the model is loaded are measured separately. Along with the average, also leave the median and slow sections. If you only did it once, write it as a single measurement and don’t decide which one is better or worse based on small differences.
Passing line for each task
Drafts read and edited by humans and values automatically entered into business systems have different tolerances for error. Errors in important fields, answers to questions that should not be answered, and behavioral changes due to injection are counted separately.
A passing score is set before the evaluation. For example, important numeric errors and creation of missing information can each be set to 0. If it doesn’t pass, in addition to changing the model, we also consider reducing the scope of the task.
reference material
Documentation and model tags checked: October 18, 2026. Primary lab environment: Windows 11 x64.
.jpg){kind=link}