Dug applies semantic web and knowledge graph methods to improve the FAIR-ness of research data.
As an example, dbGaP is a rich source of metadata about biomedical knowledge derived from clinical research like the underutilized TOPMed data sets. A key obstacle to leveraging this knowledge is the lack of researcher tools to navigate from a set of concepts of interest towards relevant study variables. In a word, search.
While other approaches to searching this data exist, our focus is semantic search: For us, "relevant" is defined as having a basis in curated, peer reviewed ontologically represented biomedical knowledge. Given a search term, Dug returns results that are related based on connections in ontological biomedical knowledge graphs.
To achieve this, we annotate study metadata with terms from biomedical ontologies, contextualize them within a unifying upper ontology allowing study data to be federated with larger knowledge graphs, and create a full text search index based on those knowledge graphs.
NOTE: You must run make init once you've cloned the repo to enable the commit-msg git hook so that conventional commits will apply automatically.
To install Dug in your environment , run make install. Alternatively,
pip install -r requirements.txt
pip install -e .To run the tests , run make test. Alternatively,
pytest .(The makefile currently only executes unit tests, whereas running pytest in the root directory will execute unit tests, integration tests, and doctests)
To bring up the backend services, run:
docker-compose upIf you're running dug-specific commands (i.e. dug) outside the docker container,
you have to make sure the env vars are set. Also, make sure all hostnames are correct
for the environment you're running in. For example, to be able to connect to dug backend
services from outside the container (but in a shell env), run:
source .env
export $(cut -d= -f1 .env)
export ELASTIC_API_HOST=localhost
export REDIS_HOST=localhost(These values are already set up in the running docker container)
Then you can actually crawl the data:
dug crawl tests/integration/data/heal_study_ddm2_HDP00166.dug.json -p "heal-ddm2"After crawling, you can search:
dug search -q "vein" -t "concepts"
dug search -q "vein" -t "variables" -k "concept=UBERON:0001638"You can also query Dug's REST API:
query="`echo '{"query" : "vein"}'`"
curl --data "$query" \
--header "Content-Type: application/json" \
--request POST \
http://localhost:5551/searchThe full set of endpoints, request/response schemas, and interactive "try it out" forms are available from the
auto-generated OpenAPI docs at http://localhost:5551/docs once the API is running. See
Search API below for an overview.
If you want to change or re-configure the dug service authentication credentials to be different from the defaults, run:
mv .env .env.bak
DATA_DIR=/path/to/dug-data-storage RANDOM=$RANDOM envsubst < .env.template > .env
docker-compose down
docker system prune -a # NOTE: This will remove *all* images, layers, and volumes.
# Be sure you're okay with this before running.
docker-compose upDug's ingest uses the Biolink upper ontology to annotate knowledge graphs and structure queries used to drive full text indexing and search. It uses Monarch Initiative APIs to perform named entity recognition on natural language prose to extract ontology identifiers. It also uses Translator normalization services to find preferred identifiers and Biolink types for each extracted identifier. The final step of ingest is to represent the annotated data in a Neo4J graph database.
Dug's integration phase uses Translator's Plater and Automat to generate a Reasoner Standard API compliant service and integrates that service into TranQL. This enables queries that span TOPMed, ROBOKOP, and other reasoners.
Dug's indexing & search phase query the graph infrastructure and analyze the resulting graphs. These are used to create documents associating natural language terms with annotations and the annotated variables and studies.
Dug will then generate Translator knowledge sources for the annotated variables and present them for query via TranQL.
Dug's parsers, indexer, and Search API all share a common set of entity types defined in the
dug-data-model package (imported as dug_data_model, currently
pinned via git+https://github.com/helxplatform/dug-data-model.git@main in requirements.txt). This gives ingest and
search a single, versioned schema (currently v2) instead of ad hoc dictionaries.
The model defines four entity types, all sharing a common base (id, name, description, type, parents,
metadata, tags, search_terms, etc.):
| Type | Description | Elasticsearch index |
|---|---|---|
DugConcept |
An ontological concept used to organize and link searchable elements. Carries identifiers (e.g. ontology CURIEs) and knowledge-graph query results (kg_answers). |
concepts_index |
DugStudy |
A study, with publications, variable_list, section_list, and an abstract. |
studies_index |
DugSection |
A CDE set / CRF (Case Report Form) grouping variables, with is_crf and variable_list. |
sections_index |
DugVariable |
A study/CDE variable, with data_type and is_cde. |
variables_index |
Index names are configurable via ELASTIC_CONCEPTS_INDEX_NAME, ELASTIC_STUDIES_INDEX_NAME,
ELASTIC_SECTIONS_INDEX_NAME, and ELASTIC_VARIABLES_INDEX_NAME (see src/dug/config.py).
Parsers (src/dug/core/parsers/) produce these typed objects directly, and the Search API's v2.0 response models
(src/dug/api_models/response_models.py) are built directly on top of the same DugConcept/DugStudy/DugSection/
DugVariable classes, so the data returned by /concepts, /studies, /cdes, and /variables mirrors the model
used during ingest.
Dug's core data structure is the knowledge graph. Here's a query of a COPDGene knowledge graph created by Dug from harmonized TOPMed variables.
Figure 1: A Biolink knowledge graph of COPDGene metadata. It shows the relationship between the biological process "Sleep" and a meta variable. The highlighted node is aTOPMed meta variable or harmonized variable. It is in turn associated with variables connected to two studies in the data set. By linking additional ontological terms to the biological process sleep, we will be able to provde progressively more helpful search results rooted in curated biomedical knowledge.
And one more example to illustrate the knowledge model we use to inform the search index:
Figure 2: The TOPMed harmonized variable is highlighted, showing its relationships with the ontology term for Heart Failure and the Heart Failure and with a specific study variable. Several similar disease, harmonized variable, variable, study relationships are also shown.
These graphs are used to create the document's well add to the search index to power full text search.
In phase 1, we use Neo4J to build queries. In subsequent phases, we integrate other semantic services using TranQL.
Figure 3: A TranQL knowledge graph query response. Integrating TOPMed harmonized variables as a Translator service called by TranQL allows us to query the federation of Translator ontological connections as a precursor to indexing. This includes chemical, phenotypic, disease, cell type, genetic, and other ontologies from sources like ROBOKOP as well as clinical aggregate data from sources like ICEES. The image above shows a query linking cholesterol to "LDL in Blood" a harmonized TOPMed variable. That variable is, in turn, linked to source variables and each of those is linked to its source study.
Elasticsearch contains the indexed knowledge graphs against which we perform full-text queries. Our query currently supports conditionals (AND, OR, NOT) as well as exact matching on quoted terms. Because we don't specify an analyzer in our query or when we index documents, we default to the standard analyzer, which is recommended for full-text search. The standard analyzer performs grammar-based tokenization (e.g., splitting up an input string into tokens by several separators including whitespace, commas, hyphens), defined by the Unicode Standard Annex #29.
Two toy examples of indexed documents, blood_color and blood_shape, are shown below to demonstrate query behavior.
The query searches the fields in all indexed documents to return a matching subset of documents.
query = {
'query_string': {
'query' : query,
'fuzziness' : fuzziness,
'fields': ['name', 'description', 'instructions', 'nodes.name', 'nodes.synonyms'],
'quote_field_suffix': ".exact"
}
}
| Query | Behavior |
|---|---|
| blood | This returns both documents (blood_color and blood_shape). |
| blood AND magenta | This returns only blood_color. |
| magenta AND cerulean | This returns blood_color even though this might be unexpected. The words 'magenta' and 'cerulean' appear in the same document in the searched fields, even though they appear in different fields, so the document is still returned. |
| blue AND square | No documents are returned. |
| blue and square | This returns both documents because the 'and' term is treated as just another term instead of an operator because it is not capitalized. The actual search resolves to blue OR and OR square |
| "round blood" | No documents are returned. |
| "blood, round" | This returns blood_shape |
| "blood round" | The document blood_shape is returned - the standard analyzer performs tokenization based on grammar separators, including commas in this case. |
The methodology, from start to finish, reads raw data, annotates it with ontological terms, normalizes those terms, inserts them into a queryable knowledge graph, queries that graph along pre-configured lines, turns the resulting knowledge graphs into documents, and indexes those documents in a full text search engine.
Link ingests raw dbGaP study metadata and performs semantic annotation by
- Parsing a TOPMed data dictionary XML file to extract variables.
- Using the Monarch SciGraph named entity recognizer to identify ontology terms.
- Using the Translator SRI identifier normalization service to
- Select a preferred identifier for the entity
- Determine the BioLink types applying to each entity
- Writing each variable with its annotations as a JSON object to a file.
- Converts the annotation format written in the steps above to a KGX graph
- Inserts that graph into a Neo4J database.
In phase-1, we query Neo4J to create knowledge graphs. In phase-2 we'll use the Neo4J to create a Translator Knowledge Provider API. That API will be integrated using TranQL with other Translator reasoners like ROBOKOP. This will allow us to build more sophisticated graphs spanning federated ontological knowledge.
- Runs those graph queries and caches knowledge graph responses.
- Consumes knowledge graphs produced by the crawl.
- Uses connections in the graph to create documents including both full text of variable descriptions and ontology terms.
- Produces a queryable full text index of the variable set.
- Presents an OpenAPI compliant REST interface
- Provides a scalable microservice suitable as an Internet endpoint.
Dug provides a tool chain for the ingest, annotation, knowledge graph representation, query, crawling, indexing, and search of datasets with metadata. The following sections provide an overview of the relevant components.
Ingest is a single step, dug crawl, which parses an input file with the given parser, annotates it with ontology
identifiers, and (via the crawl/index pipeline described below) makes it queryable through the Search API.
| Command | Description | Example |
|---|---|---|
| dug crawl | Parse, annotate, and index elements from an input file. | dug crawl {input} -p {parser} [-e {program_name}] |
-p/--parser selects which registered parser (see define_parsers in src/dug/core/parsers/__init__.py) to use
to read the input file into DugStudy/DugSection/DugVariable objects (see
The Dug Data Model); each parser knows its own input format. Currently registered parsers
include bdc, heal-ddm2, biolincc, covid19, dir, lungmap, nsrr, parent, pcgc, recover,
topmeddbgap, curesc, heartfailure, imaging, and reds.
For example, to crawl a HEAL DDM2-format study:
dug crawl tests/integration/data/heal_study_ddm2_HDP00166.dug.json -p heal-ddm2
| Command | Description | Example |
|---|---|---|
| bin/dug crawl | Execute graph queries and accumulate knowledge graphs in response. | bin/dug crawl |
| bin/dug index | Analyze crawled knowledge graphs and create search engine indices. | bin/dug index |
| bin/dug query | Test the index by querying the search engine from Python. | bin/dug query {text} |
Exposing the Elasticsearch interface to the internet is strongly discouraged for security reasons. Instead, we have a
REST API (src/dug/server.py, a FastAPI app). We use this as a place to enforce a schema and validate requests so
that the search engine's network endpoint is strictly internal. Run it with:
uvicorn dug.server:APP --port 5551Full request/response schemas are available at /docs (Swagger UI) and /redoc once the API is running — that's the
canonical reference. The current endpoints, grouped by data model type (see The Dug Data Model):
| Endpoint | Method | Description |
|---|---|---|
/concepts |
POST | Search DugConcepts related to a query, with filters/aggregations. |
/studies |
POST | Search DugStudy entities related to a query, or by parent_ids/element_ids. |
/cdes |
POST | Search DugSection (CDE set / CRF) entities. |
/variables |
POST | Search DugVariable/CDE entities, optionally scoped to parent_ids. |
/variables_by_ids |
POST | Fetch variables directly by a list of IDs. |
/study_sources |
GET | List available study sources. |
/search_program |
GET | Search for studies by program name. |
/program_list |
GET | List available programs. |
/more_like_this |
POST | Find elements similar to a given element in an index. |
/search |
POST | Legacy concept search (always queries concepts_index). |
/search_var |
POST | Legacy variable search. |
/search_var_grouped |
POST | Legacy variable search, grouped by variable ID across studies, with faceted counts. |
/search_kg |
POST | Query a cached knowledge graph for a given concept. |
/search_study |
GET | Legacy study search by ID or name. |
/dump_concepts |
POST | Dump raw indexed concepts. |
/agg_data_types |
GET | Aggregate data types across the index. |
The /concepts, /studies, /cdes, and /variables endpoints (tagged v2.0 in /docs) are the current,
data-model-backed way to query Dug; the /search* and /dump_concepts//agg_data_types endpoints predate the Dug
Data Model and are kept for backward compatibility.
A docker-compose is provided that runs four services:
- Redis
- Neo4J
- Elasticsearch
- The Dug search OpenAPI
This system can be started with the following command:
| Command | Description | Example |
|---|---|---|
| bin/dug stack | Runs all services | bin/dug stack |
Developers: Internal to bin/dug, an environment file is automatically created. That file is in docker/.env.
If you are running in development, and are not using a public IP address and hostname, you'll want to create a separate .env file to allow programs to connect to the docker containers as services. This matters if, for example, you want to run bin/test, as the clients in that test need to know how to connect to each of the services they call. Copy the generated docker/.env to docker/.env.dev. Change all hostnames to localhost. That should do it. Be sure to keep the generated passwords from the generated .env the same.
Dug's automated functional tests:
- Delete the test index
- Execute the link and load phases for the dbGaP data dictionary and harmonized variables.
- Execute the crawl and index phases.
- Execute a number of searches over the generated search index.
| Command | Description | Example |
|---|---|---|
| bin/dug test | Run automated functional tests | bin/dug test |
Once the test is complete, a command line search shows the contents of the index:
Figure 4: A command line query using the Dug Search OpenAPI to query the Elasticsearch index for a term.
The DugConcept/DugStudy/DugSection/DugVariable schema introduced in
The Dug Data Model lives in its own repository,
helxplatform/dug-data-model, and is pulled in here as the
dug_data_model package. That repository is the canonical source for:
- The schema definitions themselves (currently
v2, underdug_data_model/v2/). - Machine-readable JSON Schema and a human-readable Markdown reference generated from the models (
dug_data_model/schemas/). - A
scaffoldCLI (python -m dug_data_model.scaffold new {version}) for bootstrapping the next schema version (e.g.v3) without breaking existing consumers.
bin/export_ddm_as_json_schema.py in this repo exports the currently-pinned model version to JSON Schema for local inspection.
To release, commit the change and select feature.
During PR's several vulnerability scanners are run. If there are vulnerabilities detected, the pr checks will fail and a report will be sent to Github Security Dashboard for viewing. Please ensure the vulnerability is mitigated prior to continuing the merge to protected branches.