Added initial version of AGENTS.md#14
Conversation
This is modified version of my own agents file, works quite well on my projects.
|
@alexott I love the theme here. The hold up on the merge is the automated build scripts are failing. |
|
Maybe we don't even need it - I already submitted it to the ai-dev-kit: https://github.com/databricks-solutions/ai-dev-kit/tree/main/databricks-skills/spark-python-data-source |
|
|
| Follow this checklist (use existing sources as templates): | ||
|
|
||
| 1. Create a new folder with the project skeleton. | ||
| 2. Create new file `src/python_datasource_connectors/YourSource.py` | ||
| 3. Implement `YourSourceDataSource(DataSource)` with `name()`, `writer()`, `streamWriter()` | ||
| 4. Implement base writer class with: | ||
| - Options validation in `__init__` | ||
| - `write(iterator)` method with write logic | ||
| 5. Implement batch and stream writer classes (minimal boilerplate) | ||
| 6. Implement base reader class with: | ||
| - Options validation in `__init__` | ||
| - `read(partition)` method with read logic | ||
| - `partitions(start, end)` method to split data into partitions | ||
| 7. Implement batch and stream writer classes (minimal boilerplate) | ||
| 8. Add exports to `python_datasource_connectors/__init__.py` | ||
| 9. Create test file `tests/test_yoursource.py` with unit tests | ||
| 10. Update `README.md` with usage examples and options |
There was a problem hiding this comment.
I think the example paths in this section need to be updated after the changes we made to the repo structure?
| ### Implementation Rules | ||
|
|
||
| 1. **One concept per file**: Each module should have a single, clear purpose | ||
| 2. **Functions over classes**: Prefer functions unless you need state management |
There was a problem hiding this comment.
Will this work given that each custom source needs to extend the base data source classes?
| ### Testing | ||
| ```bash | ||
| # Run all tests | ||
| poetry run pytest | ||
|
|
||
| # Run specific test file | ||
| poetry run pytest tests/test_ds.py | ||
|
|
||
| # Run single test | ||
| poetry run pytest tests/test_ds.py::TestXxxxDataSource::test_name | ||
|
|
||
| # Run with verbose output | ||
| poetry run pytest -v | ||
| ``` | ||
|
|
||
| ### Building | ||
| ```bash | ||
| # Build wheel package | ||
| poetry build | ||
|
|
||
| # Output will be in dist/ directory | ||
| ``` | ||
|
|
||
| ### Code Quality | ||
| ```bash | ||
| # Format and lint code (ruff) | ||
| poetry run ruff check cyber_connectors/ | ||
| poetry run ruff format cyber_connectors/ | ||
|
|
||
| # Type checking | ||
| poetry run mypy cyber_connectors/ | ||
| ``` |
There was a problem hiding this comment.
Maybe we should update to the makefile commands?
4 participants
This is modified version of my own agents file, works quite well on my projects.