Task
- Inputs: contracts as ordered candidate spans (
doc_id,span_texts, or text plus character spans); a fixed set of hypotheses (key, text, short description); training contracts with annotations per (contract, hypothesis): label Entailment / Contradiction / NotMentioned and evidence-span indices. - Deliverable: one dict per requested pair, in the given order:
{"label": "Entailment" | "Contradiction", "span_scores": [float in [0, 1]] * n_spans}, scores in span order. The label is binary (requested pairs are ones the contract mentions); never output NotMentioned.
Quality
Mean over pairs of the average precision of the span scores against the gold evidence spans (mAP, higher is better). Secondary: precision at 80 % recall and label accuracy. A baseline: lexical overlap of the hypothesis' content words with the span, plus the per-hypothesis majority label.
Tools
scilib.contracts.fit_predict(train_documents, train_annotations, hypotheses, items, documents, seed=0): per-hypothesis TF-IDF logistic regression over spans (neighbour context, contract-grouped out-of-fold scores), then a LightGBM re-ranker on similarity (cosine, BM25, overlap), position and nearest-training-evidence features; a label model reads the top spans. Typed operatorcontract_evidence_identification.scilib.contracts.cross_validate(...): contract-grouped k-fold estimate (map,p_at_r80,nli_binary_accuracy,per_hypothesis_map);annotation_pairs,evaluate_predictions,average_precision,precision_at_recallfor metrics (operatorcontract_evidence_metrics).- Optional
ContractModel(plm=True)(or a subset ofPLM_GROUPS: rerank, nli, cos, emb_lr, emb_knn) adds pretrained features throughscilib.textenc: BGE-large sentence embeddings, BGE-reranker-large relevance, DeBERTa-v3-large NLI (weight idsbge_large_en,bge_reranker_large,nli_deberta_v3_large). Checkscilib.textenc.available()orscienceclaw_tools(operation=weights)first; it is costly (every span against every hypothesis; GPU or remote worker).
Rules
- Never fit or tune on the pairs being predicted; split validation folds by contract, not by pair or span.
span_scoresmust be finite, in [0, 1], one per span;labelonly Entailment or Contradiction.- Check
per_hypothesis_map: hypotheses with few training pairs fall back to a hypothesis-independent model and are weaker. - The pretrained encoders were not trained on this task's labels, but their public pretraining text may overlap contract text; disclose that when they are used.