Add Trino cost profiling support and learned default parameters - #787
Conversation
…s into TrinoOperatorsIT
|
@JunWang222 some conflicts here too |
zkaoudi
left a comment
zkaoudi
left a comment
There was a problem hiding this comment.
please check some minor comments. Thank you @JunWang222 :)
| final Job job = wayangContext.createJob(null, wayangPlan); | ||
| // ILLEGAL: We blacklist the Spark platform, although we need it. | ||
| // ILLEGAL: We blacklist all platforms that can execute the plan. | ||
| job.getConfiguration().getPlatformProvider().addToBlacklist(Java.platform()); |
There was a problem hiding this comment.
Java is added here to the black list, but only flink is added in the context (line 318). Do we need to add Java to the black list too?
There was a problem hiding this comment.
The intention of this test is to register both Java and Flink, then blacklist both executable platforms so that only the made-up platform remains and the job fails as expected. I will make this explicit by changing the context setup from makeContext(FLINK) to makeContext(BOTH).
| ```text | ||
| CPU load = alpha * number_of_rows + beta | ||
| ``` | ||
|
|
There was a problem hiding this comment.
we can add here that alpha=10 and beta=800000 in the example configuration above.
There was a problem hiding this comment.
I will add the number in the example, thank you!
|
@JunWang222 can you resolve conflicts here? |
2 participants
This PR adds Trino cost profiling support, including:
The default parameters are intended as starting values. For accurate optimization on a specific machine or deployment, users should follow the profiling guide and regenerate parameters locally.