Open
Conversation
Management command to move deployments and all related data between projects. Handles all direct and indirect relationships including Events, SourceImages, Occurrences, Jobs, Detections, Classifications, Identifications, SourceImageCollections (with mixed-collection splitting), pipeline configs, processing services, and taxa M2M links. Features: - Dry-run mode by default, --execute to commit - Per-deployment before/after snapshots with row counts - Conservation checks (source + target = original) - FK integrity and indirect access validation - Shared resource handling (clone vs reassign devices, sites, S3 sources) - Raw SQL for ProcessingService M2M to avoid ORM column mismatch Co-Authored-By: Claude <noreply@anthropic.com>
Documents the full relationship map, edge cases, and validation checklist for moving deployments between projects. Co-Authored-By: Claude <noreply@anthropic.com>
The reassign_deployments command now also recalculates: - Event cached counts (captures_count, detections_count, occurrences_count) - Both source and target project related calculated fields Co-Authored-By: Claude <noreply@anthropic.com>
- Renamed from reassign_deployments to move_project_data to better communicate the scope of the operation (all occurrences, identifications, classifications, etc. — not just deployment records) - Added conditional scope warning that shows full data weight when processed data exists, or a simple note for unprocessed transfers - Added identifier membership: users who made identifications on moved data are auto-added to the target project with Identifier role - Added default filter config copy (score threshold, include/exclude taxa, default processing pipeline) Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- Identifier users now get their existing source project role preserved (e.g. ProjectManager stays ProjectManager), falling back to Identifier role only for users who aren't source project members - TaxaLists linked to source project are now also linked to target project (M2M add, not move — TaxaLists can be shared) - Dry-run output shows TaxaLists and role assignments that will be made Co-Authored-By: Claude <noreply@anthropic.com>
✅ Deploy Preview for antenna-ssec canceled.
|
✅ Deploy Preview for antenna-preview canceled.
|
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
✅ Files skipped from review due to trivial changes (1)
📝 WalkthroughWalkthroughAdds a new Django management command Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as CLI
participant CMD as move_project_data\n(Command)
participant DB as Database
participant S3 as S3 (objects)
participant PROC as ProcessingServices
CLI->>CMD: invoke (source, deployments, target/create, flags)
CMD->>DB: validate source, deployments, collect snapshots
CMD->>DB: begin transaction
CMD->>DB: create or select target project
CMD->>DB: clone/reassign shared resources (device, site, storage)
CMD->>DB: update deployments.project_id and related FKs (events, images, occurrences, jobs)
CMD->>DB: split/clone or reassign source image collections as needed
CMD->>DB: clone pipeline configs & link processing services (raw SQL)
CMD->>DB: link taxa, taxa lists, and identifier memberships to target
CMD->>DB: run in-transaction validations (counts, FK integrity, residue checks)
CMD->>DB: commit transaction
DB-->>CMD: transaction committed
CMD->>DB: recompute cached fields and post-commit validations
CMD->>CLI: output summary, diffs, and operator notes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an operator-facing management command to move one or more deployments (and their dependent data) from a source project to a target/new project, plus a written guide documenting the relationship map and validation steps for doing so safely.
Changes:
- Added
move_project_dataDjango management command implementing deployment reassignment with dry-run analysis, execution, and post-move validation. - Added planning/ops documentation describing the full relationship map, edge cases (shared resources, mixed collections), and a validation checklist.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
| docs/claude/planning/deployment-reassignment-guide.md | New guide documenting deployment reassignment relationships, strategies, and validation checklist. |
| ami/main/management/commands/move_project_data.py | New management command to analyze and execute moving deployments and related data between projects. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Design spec for making projects portable between Antenna instances. Covers UUID fields, Organization model, natural keys, export/import commands, and Darwin Core integration. Includes 21 research areas spanning internal data validation, biodiversity standards (GBIF, iNaturalist, BOLD, Camtrap DP), and patterns from non-biodiversity applications (GitLab, WordPress, Notion, Jira). Status: draft — pending research and validation. Co-Authored-By: Claude <noreply@anthropic.com>
Documents concrete use cases for each model's UUID beyond export/import: Darwin Core field mappings (occurrenceID, eventID, etc.), ML pipeline coordination (Pipeline/Algorithm slug collision risk), device tracking across institutions, and scientific reproducibility requirements. Co-Authored-By: Claude <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@ami/main/management/commands/move_project_data.py`:
- Line 344: The code is logging user.email via self.log(...) and sending it to
logger.info, which exposes PII; update the occurrences that reference user.email
(the self.log(...) call and the logger.info usage) to avoid storing emails by
either logging a non-PII identifier (e.g., user.id or user.pk), a masked email
(e.g., mask local part), or a stable hash of the email, while preserving
role_to_assign.display_name and role_source in the message; ensure both the
self.log(...) invocation and the corresponding logger.info(...) call are changed
consistently to use the non-PII value and not user.email.
- Line 333: These lines use f-strings with no interpolation (triggering F541)
and likely loop-captured variables in inner scopes (B007); replace f-string
literals like self.log(f"\n Identifiers (users with identifications on moved
data):") with plain string literals (self.log("\n Identifiers (users with
identifications on moved data):")) and similarly update the other occurrences at
the reported locations (lines showing self.log(f"...")); search for all
self.log(f"...") instances in move_project_data.py and convert those with no
placeholders to normal strings, and where B007 arises, ensure any loop variables
used inside nested functions/comprehensions are passed as defaults or copied
into a new local variable before inner usage.
- Around line 511-515: The cloned SourceImageCollection created in
SourceImageCollection.objects.create currently only copies name, project_id and
description which drops sampling metadata; update the creation to also copy the
sampling fields from the original collection by including method and kwargs
(e.g., set method=coll.method and kwargs=coll.kwargs) so the new_coll preserves
the original collection semantics when moved.
- Around line 175-187: The command currently prefers create_project when both
--create-project and --target-project are passed; update the options handling in
move_project_data (around create_project_name, target_project logic) to detect
when both options are provided and raise a CommandError (e.g., "Specify only one
of --target-project or --create-project") instead of silently preferring
create_project; keep the existing branches for the single-option cases
(resolving target_project via Project.objects.get and logging, or setting
target_project=None and create_project_name) unchanged.
- Around line 695-723: The validation wrongly compares all records in the target
project to only the moved deployments; update the *_via_project queries
(dets_via_project, cls_via_project, idents_via_project) to also filter by
deployment_ids so they count only records for the moved deployments in the
target project (e.g. add source_image__deployment_id__in=deployment_ids to
Detection/Classifications filters and
occurrence__deployment_id__in=deployment_ids to Identification filters while
keeping source_image__project_id=target_id / occurrence__project_id=target_id).
In `@docs/claude/planning/deployment-reassignment-guide.md`:
- Around line 36-37: Update the wording so it matches the actual command
behavior: change the Device and Site guidance that currently suggests setting
their FK to NULL to instead state that the command clones or reassigns Device
and Site records (do not describe NULL as an option), and update the
project-creation description to indicate the command only sets the target
project's name and owner (omit mention of description). Ensure the edits touch
the same Device/Site lines and the project creation sentences referenced (the
blocks around the Device/Site rows and the project creation paragraph) so the
doc reflects cloned/reassigned ownership and limited project fields.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fbde3add-8c7c-4079-bf02-a2aa73a1845a
📒 Files selected for processing (3)
ami/main/management/commands/move_project_data.pydocs/claude/planning/deployment-reassignment-guide.mddocs/claude/planning/project-portability-spec.md
Fixes from Copilot and CodeRabbit review: - Mutual exclusion check for --target-project / --create-project - Pipeline config clone preserves enabled and config fields - Collection clone preserves method and kwargs fields - Validation queries scoped to moved deployments (not entire target) - Validation failure raises CommandError (triggers transaction rollback) - Project creation uses ProjectManager (create_defaults=True) - Bulk taxa linking via target_project.taxa.add(*taxa_ids) - Remove unused Taxon import and f-prefix on plain strings Add 37 automated tests covering: - Basic move, dry run, --create-project - All 6 error handling paths - Shared resource clone-vs-reassign (Device, Site, S3StorageSource) - Collection split/reassign logic - Pipeline config cloning - ProcessingService linking - Identifier role preservation - TaxaList linking, default filter copying - Edge cases (empty deployment, move all, multiple deployments) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
♻️ Duplicate comments (1)
ami/main/management/commands/move_project_data.py (1)
346-346:⚠️ Potential issue | 🟠 MajorAvoid logging identifier email addresses.
These two lines still emit
user.emailto both command output andlogger.info, which creates unnecessary PII retention for an admin migration. Loguser.pkor a masked address instead.Also applies to: 576-576
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ami/main/management/commands/move_project_data.py` at line 346, The log currently prints full PII via user.email in the self.log call; change it to use a non-PII identifier (e.g., user.pk) or a masked email instead of user.email. Update the occurrences where user.email is used in move_project_data.py (the self.log call that formats f"{user.email}: {role_to_assign.display_name} ({role_source})" and the similar occurrence around line 576) to log user.pk or a masked value while preserving role_to_assign.display_name and role_source.
🧹 Nitpick comments (1)
ami/main/tests/test_move_project_data.py (1)
325-362: Replacemsg=parameter withassertRaisesRegexto actually verify exception messages.The
msg=parameter inassertRaisesis only used when the assertion fails (i.e., no exception is raised). It does not verify the exception message contents. UseassertRaisesRegexinstead to match both the exception type and message text.Example fix
- with self.assertRaises(CommandError, msg="does not exist"): + with self.assertRaisesRegex(CommandError, "does not exist"): _run_command(*self._base_args(), "--target-project", "99999")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ami/main/tests/test_move_project_data.py` around lines 325 - 362, Replace uses of self.assertRaises(..., msg="...") with self.assertRaisesRegex(CommandError, r"pattern") in the tests listed (test_source_project_not_found, test_target_project_not_found, test_deployment_not_found, test_deployment_wrong_project, test_both_target_and_create, test_neither_target_nor_create) so the exception message is actually verified; for example call self.assertRaisesRegex(CommandError, r"Source project 99999 does not exist") around the _run_command in test_source_project_not_found and use appropriate regexes like r"does not exist", r"not found", r"not in source project", r"not both", and r"Must specify" in the corresponding tests when invoking _run_command.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@ami/main/management/commands/move_project_data.py`:
- Around line 593-603: The current copy step merges taxa and only conditionally
copies the pipeline, allowing stale values to persist; instead clear and replace
the target's defaults: call clear() on
target_project.default_filters_include_taxa and
target_project.default_filters_exclude_taxa, then add all taxa from
source_project.default_filters_include_taxa and
source_project.default_filters_exclude_taxa; always assign
target_project.default_filters_score_threshold =
source_project.default_filters_score_threshold and unconditionally set
target_project.default_processing_pipeline =
source_project.default_processing_pipeline (even if None) before saving the
project in move_project_data.
- Around line 143-166: The deployment ID parsing is brittle: parse
options["deployment_ids"] into a cleaned, deduplicated list (e.g., split on ',',
strip tokens, ignore empty tokens, attempt int() inside a try/except to raise
CommandError on non-integer values) and assign to a new variable like
deployment_ids_normalized (or replace deployment_ids); then use
Deployment.objects.filter(pk__in=deployment_ids_normalized) and compare
deployments.count() against len(deployment_ids_normalized) (not the raw parsed
list) so duplicates/trailing commas don't cause false failures; ensure any
ValueError from int() is converted into a CommandError with a clear message.
- Around line 173-189: After resolving target_project (after the block that sets
target_project from options or when create_project_name is None), add a
validation that rejects using the same project as both source and target: check
options.get("source_project") (or the earlier source_project variable if
present) against the resolved target_project.pk and raise CommandError("Target
project must be different from source project") if they match; ensure this check
runs only when target_project is not None (i.e., when --target-project was
provided).
- Line 411: The validation that raises CommandError must run inside the same
transaction.atomic() so failures roll back; move the validation logic currently
after the transaction.atomic() block (the block starting at the
transaction.atomic() call and ending at that block's closing) into the atomic
block before it exits—specifically relocate the validation checks (the section
that raises CommandError) into the transaction.atomic() scope in
move_project_data.py so any CommandError thrown triggers a rollback; ensure all
subsequent writes and the commit point occur only after validation completes
successfully.
---
Duplicate comments:
In `@ami/main/management/commands/move_project_data.py`:
- Line 346: The log currently prints full PII via user.email in the self.log
call; change it to use a non-PII identifier (e.g., user.pk) or a masked email
instead of user.email. Update the occurrences where user.email is used in
move_project_data.py (the self.log call that formats f"{user.email}:
{role_to_assign.display_name} ({role_source})" and the similar occurrence around
line 576) to log user.pk or a masked value while preserving
role_to_assign.display_name and role_source.
---
Nitpick comments:
In `@ami/main/tests/test_move_project_data.py`:
- Around line 325-362: Replace uses of self.assertRaises(..., msg="...") with
self.assertRaisesRegex(CommandError, r"pattern") in the tests listed
(test_source_project_not_found, test_target_project_not_found,
test_deployment_not_found, test_deployment_wrong_project,
test_both_target_and_create, test_neither_target_nor_create) so the exception
message is actually verified; for example call
self.assertRaisesRegex(CommandError, r"Source project 99999 does not exist")
around the _run_command in test_source_project_not_found and use appropriate
regexes like r"does not exist", r"not found", r"not in source project", r"not
both", and r"Must specify" in the corresponding tests when invoking
_run_command.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 21aec0e3-2104-4d9f-abb2-c75687561306
📒 Files selected for processing (3)
ami/main/management/commands/move_project_data.pyami/main/tests/__init__.pyami/main/tests/test_move_project_data.py
- Move all validation checks inside transaction.atomic() so failures trigger a full rollback instead of leaving partially-moved data - Add source==target project guard (prevents self-move corruption) - Deduplicate --deployment-ids at parse time - Remove PII (email addresses) from log output, use user IDs instead - Update deployment-reassignment-guide to match shared-vs-exclusive logic - Add tests for source==target and duplicate deployment IDs (39 total) Co-Authored-By: Claude <noreply@anthropic.com>
Resolves Django test loader collision with the new ami/main/tests/ package (added for test_move_project_data). Both can't coexist — loader raises 'tests module incorrectly imported'. Co-Authored-By: Claude <noreply@anthropic.com>
…tep numbers - Pull S3StorageSource/Device/Site clone-vs-reassign loop into a single module-level helper. Each call site collapses to ~5 lines. - Replace numeric `[N/16]` step prefixes with semantic labels (`[create-target]`, `[move-deployment]`, etc). Numbering had drifted to two ranges and 5 different ops shared `[10/12]`, which made log bisection painful. Labels are stable when steps are added or removed. Co-Authored-By: Claude <noreply@anthropic.com>
When a SourceImageCollection is mixed (referenced by both moved and staying deployments), the script clones it into the target project. Previously the Job's `source_image_collection_id` was untouched, so a moved Job was still pointing at the source-project collection — re-running the Job would have processed the wrong dataset. - Track `collection_clone_map` (old_pk -> new_pk) during the split step. - Remap moved Jobs to the cloned target collection PKs. - For `--no-clone-collections`, null out the FK on moved Jobs instead of letting them cross-link (Job loses its rerun target, which is documented in the warn log). - Surface DataExport residue in the log: moved Jobs whose `data_export.project_id` still points at source. Historical artifacts; not migrated. - Validation: fail+rollback if any moved Job ends up cross-linked to a SourceImageCollection on a non-target project. - Tests: assert remap on mixed-split, and that --no-clone nulls instead of leaving cross-link. Co-Authored-By: Claude <noreply@anthropic.com>
ProjectSettingsMixin defines five settings fields. The default-filter copy block was only carrying four (score threshold, include/exclude taxa, default_processing_pipeline). `session_time_gap_seconds` controls Event grouping for new captures and was silently inheriting the default (2h) on a newly-created target instead of the source's value. Co-Authored-By: Claude <noreply@anthropic.com>
…triggering Bulk .update() bypasses post_save on Event/SourceImage/Occurrence/Job. The script's compensation only covers cached-count fields. There are no post_save listeners on those models in the codebase today, but if any are added later (search index, audit log, cache warmer) they would silently miss moved data. `--fire-post-save` calls full Deployment.save() after commit, which fires post_save on each moved deployment AND queues Celery event regrouping via regroup_async=True. Off by default to keep big moves fast. Co-Authored-By: Claude <noreply@anthropic.com>
Surface non-obvious post-move state directly in command output rather than expecting operators to read PR descriptions or session notes: - Cache lag: cachalot and UI may show stale counts briefly after commit - Signal status: whether post_save fired (and what it implies) - DataExport residue: count of moved Jobs whose data_export still points at another project (historical artifacts; intentionally left in place) - Source-side residue: shared resources cloned, TaxaLists shared, taxa M2M copied — clarifies that the source project is not gutted of these These are the things a human re-reading the log a week later would otherwise have to dig out of code or docs. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Management command to move deployments and all associated data from one project to another. Motivated by a request to move northern Quebec stations (Umiujaq, Kuujjuaq, Kangiqsujuaq, Inukjuak) from "Insectarium de Montreal" into a new "Nunavik" project.
This is a comprehensive data transfer that handles 16 categories of related data:
Usage
Features
Fixes from review
--target-project/--create-projectenabledandconfigfieldsmethodandkwargsfieldsCommandError(rolls back transaction)Project.objects.create(create_defaults=True)via ProjectManagertarget_project.taxa.add(*taxa_ids)Taxonimport and f-prefix on strings without placeholdersTakeaway-review fixes (2026-05-01)
SourceImageCollectionwas mixed (referenced by both moved and staying deployments), the script cloned it to the target project but moved Jobs still pointed at the source collection. Now:collection_clone_mapis built during the split step and moved Jobs are remapped to the cloned target collection. With--no-clone-collections, the FK is nulled instead so no cross-link survives.session_time_gap_secondsis now copied to the target project (was the oneProjectSettingsMixinfield missing from the default-filter copy block).--fire-post-saveflag added. Off by default. Bulk.update()bypassespost_saveon Event/SourceImage/Occurrence/Job; no listeners exist on those models today, but the flag is the opt-in path if any are added later. Calls fullDeployment.save()post-commit (also queues Celery event regrouping).clone_or_reassignhelper collapses 3× S3/Device/Site loops.[create-target],[move-deployment], ...) — old[N/12]and[N/16]ranges were both used and 5 different ops shared[10/12].Testing & validation
Automated tests (41 tests)
Test file:
ami/main/tests/test_move_project_data.pydocker compose -f docker-compose.ci.yml run --rm django python manage.py test ami.main.tests.test_move_project_data --keepdb -v2TestMoveToExistingProjectTestDryRun--executeTestCreateProject--create-projectflag, member copyingTestErrorHandlingTestSharedResourceCloningTestCollectionHandling--no-clone-collections, mixed-split remaps moved Jobs,--no-clone-collectionsnulls moved Job collection FKTestPipelineConfigCloning--no-clone-pipelinesTestProcessingServiceLinkingTestIdentifierRolePreservationTestTaxaListLinkingTestDefaultFilterCopyingTestEdgeCasesManual validation (pre-production)
Before running on production data, validate on a staging DB or local backup:
Dry run first — review the full output, check per-deployment counts match expectations
python manage.py move_project_data --source-project 23 --create-project "Nunavik" --deployment-ids 84,163,284Execute — review the validation output at the end
python manage.py move_project_data --source-project 23 --create-project "Nunavik" --deployment-ids 84,163,284 --executePost-move spot checks in Django admin or shell:
If validation fails, the transaction is rolled back automatically and a
CommandErroris raised with details. No manual cleanup needed.Test plan
Summary by CodeRabbit
New Features
Documentation
Tests