Computer Vision: Histopathological Images

Shi et al. (2020) conducted a systematic study on the effects of annotation granularity in deep learning models for histopathological images. They tested four levels of annotation detail:

1. Image-wise: A single label for the entire image (coarse)
2. Bounding box: Rectangular boxes around regions of interest
3. Ellipse-wise: Elliptical annotations following cell shapes
4. Pixel-wise: Precise segmentation masks (fine)

The results were clear. For classification tasks, state-of-the-art deep learning classifiers performed significantly better when trained on pixel-wise annotated datasets. On average, precision improved by 7.87%, recall by 8.83%, and F1-score by 7.85%. For segmentation tasks, semantic segmentation algorithms achieved 8.33% better accuracy when trained with pixel-wise annotations compared to coarser alternatives.

‍

The study concluded that "finer-grained annotation can improve the performance of deep learning models, but also help extracts more accurate phenotypic information from histopathological slides." This matters for medical applications where extracted features inform diagnosis and treatment decisions. Intelligence systems trained on granular annotations can help pathologists inspect specific regions for better diagnosis.

‍

Medical Imaging: The Shortcut Learning Problem

Performance on internal test sets tells only part of the story. Luo et al. (2022), publishing in Radiology: Artificial Intelligence, investigated whether annotation granularity affects model generalizability in chest radiograph diagnosis. Their findings revealed a critical weakness in models trained on coarse annotations.

The research team compared two approaches:

CheXNet: Trained on radiograph-level annotations (coarse). The model received a single label per image indicating the presence or absence of diseases like pneumonia, cardiomegaly, or nodules.

CheXDet: Trained on fine-grained lesion bounding boxes. Annotators marked the specific location of pathological findings on each radiograph.

Both models achieved radiologist-level performance on internal test data. This seemed to validate the coarse annotation approach. However, when tested on external datasets from different medical centers (NIH ChestX-ray14 and PadChest), CheXNet's performance degraded significantly while CheXDet maintained accuracy.

‍

The explanation lay in shortcut learning. Models trained on radiograph-level annotations learned to rely on unintended patterns rather than true pathologic signs. Gradient-weighted class activation maps (Grad-CAM) revealed that CheXNet often attended to false-positive regions or missed lesions entirely, yet still made correct predictions on internal data by exploiting dataset-specific artifacts.

‍

Fine-grained annotations overcame this problem. By forcing the model to localize lesions during training, CheXDet learned to identify correct pathologic patterns. This made it more robust to external data and less prone to shortcut learning. The study found that CheXDet achieved higher external performance than CheXNet without loss of internal accuracy. For small lesions like nodules and masses, CheXDet trained on only 20% of the data outperformed CheXNet trained on 100% of the data.

‍

The authors concluded: "Our findings highlight the importance of using fine-grained annotations for developing trustworthy DL-based medical image diagnoses." They also noted that "the claim that DL demonstrates performance similar to that of physicians may need further investigation" when models rely on shortcuts rather than genuine diagnostic reasoning.

‍

Natural Language Processing: The Quantity-Quality Tradeoff

While medical imaging provides clear evidence for fine-grained annotations, natural language processing presents a more nuanced picture. Mallen and Belrose (2024) explored the microeconomics of the quantity-quality tradeoff in binary NLP classification tasks. Their work focused on a practical reality: neither label quantity nor quality is fixed. Practitioners face budget constraints and must decide how to allocate resources between collecting more labels and improving label quality.

‍

The research identified three distinct regimes:

• Quantity-dominant regime: When the model lacks basic knowledge of the task, more low-quality labels provide better performance than fewer high-quality labels. The model benefits from exposure to more examples, even if those examples are imperfect.

• Quality-dominant regime: Once the model has learned basic patterns, label quality becomes more important than quantity. Fewer high-quality labels outperform more low-quality labels because they correct the model's errors and refine its understanding

• Mixed regime: The optimal strategy often involves combining low-quality and high-quality data. This achieves higher accuracy at lower cost than using either alone. A small set of carefully annotated examples guides the model, while a larger set of noisier labels provides coverage.

The study established a Pareto frontier of scalable elicitation methods that optimally trade off labeling cost and classifier performance. They found that supervised fine-tuning accuracy could be improved by up to 5 percentage points at a fixed labeling budget by adding a few-shot prompt to leverage the model's existing knowledge.

‍

This research suggests that annotation depth decisions should consider the model's current state. Early in development, quantity may matter more. As the model matures, quality becomes critical. The most cost-effective approach often mixes annotation depths strategically.

‍

Cost Optimization: Making Fine-Grained Annotation Feasible

The performance benefits of fine-grained annotations are clear, but cost remains a barrier. Lawley et al. (2024), writing in Biomedical Signal Processing and Control, developed a cost-focused framework for optimizing data collection and annotation in medical ultrasound imaging. Their approach borrows methods from clinical trial design to quantify how much data and annotation are required to achieve specific research objectives.

‍

The framework operates in two phases:

Phase 1: Iterative Accuracy Prediction. The team trains models on progressively larger subsets of the data and measures accuracy at each step. This generates a learning curve that predicts how accuracy will improve with additional data. The relationship follows a predictable pattern, with the majority of accuracy improvements occurring using only 40-50% of the available data, depending on the tolerance measure.

‍

Phase 2: Active Learning Optimization. Rather than annotating all remaining data, active learning identifies which samples will most improve the model. The algorithm selects examples where the model is most uncertain or where annotation is expected to provide the greatest information gain. This reduces the amount of manual annotation required.

‍

The results demonstrated substantial cost savings. Manual annotation could be reduced by 66% while accepting only a 4% accuracy drop from theoretical maximums. This makes fine-grained annotation economically feasible for projects with fixed budgets.

‍

The significance of this work lies in its ability to quantify annotation requirements. Rather than guessing how much data is needed, project managers can predict the accuracy-sample size relationship and make informed decisions about resource allocation. These methods are already well understood by clinical funders, providing a valuable framework for feasibility and pilot studies where machine learning will be applied within budget constraints

The optimal annotation depth varies by task type. Different machine learning problems have different sensitivities to annotation granularity.

‍

Object Detection and Segmentation

Computer vision tasks that require spatial localization benefit strongly from fine-grained annotations. Object detection needs bounding boxes at minimum. Instance segmentation requires pixel-level masks. The Shi et al. study demonstrated that segmentation algorithms achieve 8.33% better accuracy with pixel-wise annotations compared to coarser alternatives.

‍

However, the level of precision required depends on the application. Autonomous vehicles need precise segmentation of road boundaries, pedestrians, and vehicles because positioning errors can cause accidents. A content moderation system that flags inappropriate images may only need image-level labels because the exact location of problematic content is less critical.

‍

Named Entity Recognition

Named entity recognition (NER) in natural language processing requires span-level annotations. Annotators must mark the exact boundaries of entities (names, locations, organizations) in text. This is inherently a fine-grained task. Coarser sentence-level or document-level labels cannot provide the supervision needed for token-level predictions.

‍

The depth question in NER relates to entity type granularity. A coarse scheme might label all organizations the same way. A fine-grained scheme distinguishes companies, government agencies, non-profits, and educational institutions. Research on annotation granularity in clinical NLP has shown that finer entity type distinctions improve downstream task performance when those distinctions are clinically meaningful, but add noise when they are not.

‍

Sentiment Analysis

Sentiment analysis presents a spectrum of annotation depths. Document-level sentiment (positive, negative, neutral) is the coarsest approach. Sentence-level sentiment captures more nuance. Aspect-based sentiment annotation identifies specific features or aspects mentioned in text and labels sentiment toward each aspect separately. Target-dependent sentiment goes further, annotating sentiment toward specific entities mentioned in context.

‍

The choice depends on the application. A movie review classifier may only need document-level sentiment. A product feedback analysis system benefits from aspect-based annotations that distinguish sentiment toward price, quality, customer service, and features. The additional annotation cost is justified when the business needs that level of detail.

‍

Classification with Rationales

An emerging approach in annotation design asks annotators to provide rationales for their labels. Rather than just marking a document as positive or negative, annotators highlight the specific words or phrases that influenced their decision. This creates a form of fine-grained annotation without requiring a complete restructuring of the task.

‍

Research on machine learning with annotator rationales has shown that rationales help models learn more efficiently. They reduce the amount of labeled data needed to achieve target performance. This approach is particularly valuable when annotation budget is limited but annotators can provide rationales with minimal additional effort.

‍

Domain-Specific Requirements

Beyond task type, domain considerations influence annotation depth decisions.

‍

Healthcare and Medical AI

Healthcare applications demand fine-grained annotations for several reasons. First, model errors have serious consequences. A misdiagnosis or missed finding can harm patients. Models must learn correct diagnostic reasoning, not shortcuts. The Luo et al. study on chest radiographs demonstrated that fine-grained annotations prevent shortcut learning and improve generalizability.

‍

Second, regulatory requirements for medical AI systems increasingly emphasize explainability and trustworthiness. The U.S. Food and Drug Administration's guidance on clinical decision support software considers how AI systems make decisions. Models that rely on shortcuts or unintended patterns face regulatory scrutiny. Fine-grained annotations that force models to attend to clinically relevant features align with regulatory expectations.

‍

Third, medical annotation often requires expert domain knowledge. Radiologists, pathologists, and other specialists are expensive annotators. This creates pressure to minimize annotation volume. The cost-optimization framework from Lawley et al. shows how active learning can reduce expert annotation requirements by 66% while maintaining acceptable accuracy. This makes fine-grained medical annotation economically viable.

‍

Legal and Compliance

Legal document analysis presents similar considerations. Contract review, e-discovery, and regulatory compliance tasks require high accuracy because errors create legal risk. Fine-grained annotations that mark specific clauses, obligations, and risks help models learn precise legal reasoning.

‍

However, legal annotation faces a different cost structure than medical annotation. Law firms and legal departments often have paralegals and junior associates who can perform annotation at lower cost than senior attorneys. This shifts the economic calculation. Volume of annotation becomes more feasible, but quality control becomes critical. Inter-annotator agreement metrics and expert review processes ensure annotation consistency.

‍

Customer Intelligence and Marketing

Commercial applications like customer feedback analysis, brand monitoring, and market research operate under different constraints. Model errors rarely have serious consequences. The business value comes from aggregate insights across thousands or millions of examples, not perfect accuracy on individual cases.

‍

This suggests coarser annotations may suffice. Document-level sentiment or topic labels provide enough signal for trend analysis and dashboard reporting. The cost savings from coarse annotation allow processing larger volumes of data, which may provide more business value than perfect accuracy on smaller datasets.

‍

However, some commercial applications benefit from fine-grained annotations. Aspect-based sentiment analysis helps product teams understand specific features customers like or dislike. This requires more detailed annotation but delivers more actionable insights. The decision depends on how the insights will be used and whether the additional detail justifies the cost.