feat: add benchmark for graphql-tools-mocking - #5
Open
alias-mac wants to merge 6 commits into
Open
Conversation
Add information about issues with arguments.
This will allow us to test the performance of the system with different schema sizes.
This is for graphql-tools-mocking setup.
Specs: MacBook Pro (16-inch, 2021) Chip Apple M1 Pro Memory 32 GB
Specs: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz Memory 64 GB
alias-mac
commented
Sep 14, 2022
| @@ -0,0 +1,5 @@ | |||
| | Command | Mean [s] | Min [s] | Max [s] | Relative | | |||
Contributor
Author
There was a problem hiding this comment.
I know these generated files will cause issues with linting, but I'm not sure if we want to commit them or not.
On one hand it will be easier to reference/point to the data, on the other it is generated and thus not needed to be part of the repo.
hjdivad
reviewed
Oct 14, 2022
Comment on lines
+38
to
+42
| $ hyperfine --warmup 3 -r 10 \ | ||
| 'ts-node -T graphql-tools-mocking/benchmark.ts graphql-tools-mocking.graphql' \ | ||
| 'ts-node -T graphql-tools-mocking/benchmark.ts <api-1>.graphql' \ | ||
| 'ts-node -T graphql-tools-mocking/benchmark.ts <api-2>.graphql' \ | ||
| --export-markdown benchmark-result-<architecture>-<with-server/without-server>.md |
Contributor
There was a problem hiding this comment.
What exactly are we measuring? Is it time to boot + mock 1 query?
Contributor
Author
There was a problem hiding this comment.
yes, in this case it is warm boot tho. The point here was not to measure the performance of mocking 1 query, but test the performance impact of big schemas and, as they grow, will this cause problems for our development productivity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will allow us to test the performance of the system with different
schema sizes.
Benchmarks
Benchmarks attached in the PR for ts-web and v-web on multiple scenarios/environments.
There isn't much difference on booting with GQL server or without, thus it shouldn't be a decision maker.
Documentation added on how to run these benchmarks.
Linux benchmarks:
ts-node -T graphql-tools-mocking/benchmark.ts graphql-tools-mocking.graphqlts-node -T graphql-tools-mocking/benchmark.ts ts-deco-fe.federated.graphqlts-node -T graphql-tools-mocking/benchmark.ts voyager-api.federated.graphqlMacbook Pro with M1 Pro CPU:
ts-node -T graphql-tools-mocking/benchmark.ts graphql-tools-mocking.graphqlts-node -T graphql-tools-mocking/benchmark.ts ts-deco-fe.federated.graphqlts-node -T graphql-tools-mocking/benchmark.ts voyager-api.federated.graphql