beita6969 / knowledge-discovery

Knowledge Discovery & Graphs

Purpose

Discover hidden patterns, build knowledge graphs, and extract novel insights from structured and unstructured data.

Key Datasets

• WALS (wals.info): World Atlas of Language Structures — 192 linguistic features across 2,679 languages in CLDF format (CC-BY 4.0)
• HistWords (nlp.stanford.edu/projects/histwords): Historical word embeddings tracking semantic change across 4 languages over centuries (.npy/.pkl format)

Protocol

1. Data exploration — Profile data, identify patterns, check distributions
2. Feature engineering — Create derived features, temporal features, cross-references
3. Pattern detection — Apply clustering, association rules, anomaly detection
4. Knowledge graph construction — Build entity-relation graphs from discovered patterns
5. Insight generation — Interpret patterns in domain context
6. Validation — Verify discoveries against known phenomena

Discovery Types

• Linguistic typology: Cross-linguistic universals, language family features, areal patterns
• Semantic change: Word meaning evolution, neologism tracking, conceptual drift
• Scientific trends: Emerging research topics, citation patterns, collaboration networks
• Biomedical discovery: Drug repurposing candidates, gene-disease associations

Rules

• Distinguish between correlation and causation in discovered patterns
• Report statistical significance and effect sizes
• Validate against domain expertise and existing literature
• Handle missing data transparently
• For knowledge graphs, use standard ontologies (RDF, OWL) when possible