fix(schema): pathlib.Path is treated as cog.Path; aliased cog types fail - #3171
Open
BarneyChambers wants to merge 5 commits into
Open
fix(schema): pathlib.Path is treated as cog.Path; aliased cog types fail#3171BarneyChambers wants to merge 5 commits into
BarneyChambers wants to merge 5 commits into
Conversation
PrimitiveFromName matched the local identifier Path, so pathlib.Path got format:uri in the OpenAPI schema while the runtime never downloaded the file. from cog import Path as CogPath failed for the same reason. Signed-off-by: barneychambers <barneychambers@hotmail.com>
The same message fires for return annotations, so calling them inputs was wrong. Signed-off-by: barneychambers <barneychambers@hotmail.com>
Outputs already upload os.PathLike, so failing the build for `-> pathlib.Path` was a regression. Relative re-exports like `from .types import Path` are left as file URIs. Signed-off-by: barneychambers <barneychambers@hotmail.com>
Docs still said PathLike inputs were cog.Path. They are not: coglet only downloads cog.types.Path. Say that, reject pathlib.Path even without an import, and follow local `from .types import Path` when the file re-exports pathlib. Signed-off-by: barneychambers <barneychambers@hotmail.com>
from .types import Path was followed, but import helpers then helpers.Path still emitted a file URI. Record Path/File/Secret on local module imports, including helpers/__init__.py packages. Signed-off-by: barneychambers <barneychambers@hotmail.com>
Author
|
@replicate/cog this is ready for review. Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cog builds an OpenAPI schema from the predictor's type hints at
cog buildtime. File inputs are supposed to becog.Path. Those show up in the schema as{type: string, format: uri}, and at runtime Cog downloads the URL to a temp file before callingpredict().The schema generator does not check where
Pathwas imported from. Any annotation namedPathbecomes a file-URI field.So a normal pathlib import makes the schema lie for inputs, and the obvious workaround (
Path as CogPath) does not compile.File outputs are different. The worker already uploads anything
os.PathLike, includingpathlib.Path. Those models work today. This change does not fail the build for-> Path.Real world example
Lots of models already import pathlib, then annotate the file input with
Path:The author wanted a downloaded file. They got stdlib
pathlib.Path.What happens on main today:
Pathand emits{type: string, format: uri}.cog buildsucceeds.http://...ordata:...URL. Schema validation accepts it.cog.types.Path. This one is not, so no download.predict()receives the URL string.image.read_text()raises, or the model treats a URL as a filesystem path.The image looks fine. The prediction is what breaks.
People then try to keep pathlib and alias the cog type:
That fails schema generation too:
CogPathis reported as an unknown external type, even though it iscog.Pathunder an alias.After this change: an input
from pathlib import Pathfails the build with a message to usefrom cog import Path.from cog import Path as CogPathis treated as a real file input (format: uri). A return type ofpathlib.Pathstill builds, same as today.Fix
Path/File/Secretthrough the import (from cog import Path as CogPathisPathfromcog).from pathlib import Path,pathlib.Path,import pathlib as pthenp.Path).from .types import Path, andimport helpersthenhelpers.Path(includinghelpers/__init__.py). pathlib if that file imported Path from pathlib, cog.Path if from cog.pathlib.Pathas a file URI. The worker already uploadsos.PathLike.Pathwith no import still counts ascog.Path, same as today.from pathlib import *is the same leftover: the import table never records*. Unresolved relative imports (no file on disk) stay file URIs.Docs
docs/python.mdpreviously said anyos.PathLikesubclass was accepted as an input and treated ascog.Path. That does not match coglet, which only downloadscog.types.Path. The docs now requirefrom cog import Pathfor file inputs, and say a return type ofpathlib.Pathis still fine.Not a duplicate
#3094 is about rejecting
Input(default=Path("image.png")). Different bug: that one is defaults. This one is the type name resolving to the wrong primitive.Test
Nothing on main covered
pathlib.Pathorfrom cog import Path as CogPath. Existing Path tests all usefrom cog import Path. Added:TestAliasedCogPathInputTestAliasedCogPathOpenAPIIsURITestAliasedCogSecretInputTestCogTypesPathImportTestQualifiedCogPathInputTestPathlibPathInputRejectedTestPathlibPathOutputAcceptedTestRelativeTypesPathInputTestQualifiedPathlibPathRejectedTestQualifiedPathlibPathWithoutImportRejectedTestPathlibImportedAsAliasRejectedTestAliasedCogPathAlongsidePathlibTestRelativePathlibPathInputRejectedTestRelativeCogPathReexportAcceptedTestImportedModulePathlibPathRejectedTestImportedModulePathlibPathAliasedRejectedTestImportedModuleCogPathAcceptedTestImportedPackagePathlibPathRejectedThey fail on
main(except the output / unresolved-re-export / cog-reexport cases, which pass on both) and pass with this change.This changes primitive resolution in
pkg/schemaand follows Path/File/Secret through local modules after they are loaded. The runtime already checkscog.types.Pathby identity for downloads, which is correct. The schema was the part that was wrong on inputs.Verification
from cog import Path as CogPathproducesformat: uri.from pathlib import Pathfails the build.pathlib.Pathfails the build even withoutimport pathlib.from .types import Pathandimport helpers/helpers.Pathfail when the local module re-exports pathlib, and succeed when it re-exports cog.Path.pathlib.Pathstill builds.from cog import Pathandcog.Pathstill work.cog.Path; pathlib.Path is still valid as a return type.