Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 3 additions & 16 deletions services/data-ngin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,6 @@ The **data-ngin** is a modular pipeline designed to fetch, clean, store, and ana
- **Modularity:** Designed with interchangeable components for fetchers, loaders, cleaners, and inserters

## Project Structure
- **`src/main.py`**: Entry point for pipeline execution
- **Primary Modules**:
- **Loader**: Loads metadata and configuration (e.g., `CSVLoader`)
- **Fetcher**: Fetches raw data (e.g., `DatabentoFetcher`)
Expand All @@ -62,7 +61,6 @@ The **data-ngin** is a modular pipeline designed to fetch, clean, store, and ana
├── src
│ ├── config
│ │ ├── config.yaml
│ ├── main.py
│ ├── modules
│ │ ├── cleaner
│ │ │ ├── cleaner.py
Expand Down Expand Up @@ -122,7 +120,7 @@ The **data-ngin** is a modular pipeline designed to fetch, clean, store, and ana

1. **Clone the Repository:**
```bash
git clone https://github.com/AlgoGators/data-ngin.git
git clone https://github.com/AlgoGators/algogators.git
cd data-ngin
```

Expand Down Expand Up @@ -159,11 +157,6 @@ The **data-ngin** is a modular pipeline designed to fetch, clean, store, and ana

## Usage

### Run Pipeline Locally
```bash
poetry run python src/main.py
```

### Access Airflow Web Interface
1. Navigate to http://localhost:8080
2. Use default credentials (admin/admin) unless modified
Expand Down Expand Up @@ -435,19 +428,13 @@ task = PythonOperator(task_id="fetch_data", python_callable=orchestrator.run, da

## Common Development Tasks

1. **Run the pipeline manually**

```bash
poetry run python src/main.py
```

2. **Check database contents**
1. **Check database contents**

```sql
SELECT * FROM futures_data.ohlcv_1d LIMIT 10;
```

3. **Trigger Airflow DAG**
2. **Trigger Airflow DAG**

```bash
airflow dags trigger data_pipeline
Expand Down
45 changes: 0 additions & 45 deletions services/data-ngin/src/data_ngin/main.py

This file was deleted.

Loading