Skip to content

Issue #4189 : SQL File Output, option to skip INSERT statements and select fields #8782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bamaer merged 1 commit into from
Oct 9, 2026
Merged

Issue #4189 : SQL File Output, option to skip INSERT statements and select fields #8782

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/sqlfileoutput.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ The information in the connection is used to select the appropriate dialect.
* Add create table statement : check this if you want the "CREATE TABLE" statement to be included in the generated SQL
* Add truncate table statement: check this if the table is expected to exist and you want the "TRUNCATE TABLE" statement to be included in the generated SQL.
Note that this could also be a "DELETE FROM" statement, depending on the database dialect.
* Do not add INSERT statements : only available when "Add create table statement" is checked.
Enable this to generate only the CREATE TABLE statement, without an INSERT statement for every input row.
* Start new line for each statement : Enable this to increase the readability of the file
* Filename : the filename (without file extension)
* Create parent folder : if you want the parent folder of the specified file to be created if it doesn't exist already.
Expand All @@ -68,4 +70,10 @@ A unique list is being kept in memory that can be used in the next workflow acti
|Option|Description
|Date format|Allows you to specify the date format.
|Encoding|Allows you to specify the text file encoding
|===
|Specify table fields|Enable this to only use the fields listed below in the generated CREATE TABLE and INSERT statements. When disabled, all incoming fields are used.
|Fields a|
* Name : the name of the incoming field to include
* Rename : optional new column name to use in the generated SQL

Use the "Get fields" button to fill the grid with the fields of the previous transform.
|===
289 changes: 289 additions & 0 deletions integration-tests/database/0029-sql-file-output-selected-fields-validate-file.hpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--

Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

-->
<pipeline>
<info>
<pipeline_version/>
<capture_transform_performance>N</capture_transform_performance>
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<pipeline_status>0</pipeline_status>
<parameters/>
<name>0029-sql-file-output-selected-fields-validate-file</name>
<name_sync_with_filename>Y</name_sync_with_filename>
<description/>
<extended_description/>
<created_user>-</created_user>
<modified_user>-</modified_user>
<created_date>2026/10/06 14:12:52.063</created_date>
<modified_date>2026/10/06 14:12:52.063</modified_date>
<created_hop_version/>
<modified_hop_version>2.20.0-SNAPSHOT</modified_hop_version>
</info>
<transform>
<type>Abort</type>
<name>Abort</name>
<row_threshold>0</row_threshold>
<always_log_rows>Y</always_log_rows>
<abort_option>ABORT_WITH_ERROR</abort_option>
<distribute>Y</distribute>
<copies>1</copies>
<GUI>
<xloc>704</xloc>
<yloc>208</yloc>
</GUI>
<description/>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<attributes/>
</transform>
<transform>
<type>DataGrid</type>
<name>Data grid</name>
<fields>
<field>
<currency/>
<decimal/>
<group/>
<name>sql_file</name>
<type>String</type>
<format/>
<length>-1</length>
<precision>-1</precision>
<set_empty_string>N</set_empty_string>
</field>
</fields>
<data>
<line>
<item>(</item>
</line>
<line>
<item> col1 TEXT</item>
</line>
<line>
<item>)</item>
</line>
<line>
<item>;</item>
</line>
<line>
<item/>
</line>
<line>
<item>INSERT INTO "public".testtable(col1) VALUES (E'this');</item>
</line>
<line>
<item>INSERT INTO "public".testtable(col1) VALUES (E'some');</item>
</line>
</data>
<distribute>Y</distribute>
<copies>1</copies>
<GUI>
<xloc>192</xloc>
<yloc>112</yloc>
</GUI>
<description/>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<attributes/>
</transform>
<transform>
<type>FilterRows</type>
<name>Filter rows</name>
<compare>
<condition>
<negated>N</negated>
<operator>-</operator>
<leftvalue>sql_file</leftvalue>
<function>&lt;></function>
<rightvalue>content</rightvalue>
<conditions/>
</condition>
</compare>
<distribute>Y</distribute>
<copies>1</copies>
<GUI>
<xloc>512</xloc>
<yloc>208</yloc>
</GUI>
<description/>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<attributes/>
</transform>
<transform>
<type>MergeJoin</type>
<name>Merge join</name>
<join_type>FULL OUTER</join_type>
<transform1>Data grid</transform1>
<transform2>Text file input</transform2>
<keys_1>
<key>sql_file</key>
</keys_1>
<keys_2>
<key>content</key>
</keys_2>
<distribute>Y</distribute>
<copies>1</copies>
<GUI>
<xloc>352</xloc>
<yloc>208</yloc>
</GUI>
<description/>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<attributes/>
</transform>
<transform>
<type>TextFileInput2</type>
<name>Text file input</name>
<file_type>CSV</file_type>
<compression>None</compression>
<separator>[]]';''./</separator>
<enclosure/>
<enclosure_breaks>N</enclosure_breaks>
<escapechar/>
<null_if_not_enclosed>N</null_if_not_enclosed>
<header>Y</header>
<prependFileName>N</prependFileName>
<nr_headerlines>1</nr_headerlines>
<footer>N</footer>
<nr_footerlines>1</nr_footerlines>
<line_wrapped>N</line_wrapped>
<nr_wraps>1</nr_wraps>
<layout_paged>N</layout_paged>
<nr_lines_per_page>80</nr_lines_per_page>
<nr_lines_doc_header>0</nr_lines_doc_header>
<noempty>Y</noempty>
<include>N</include>
<include_field/>
<rownum>N</rownum>
<rownum_field/>
<rownumByFile>N</rownumByFile>
<format>mixed</format>
<encoding/>
<limit>0</limit>
<date_format_lenient>Y</date_format_lenient>
<date_format_locale>en_US</date_format_locale>
<length>Characters</length>
<filters/>
<error_count_field/>
<error_fields_field/>
<error_text_field/>
<error_line_skipped>N</error_line_skipped>
<schema_definition/>
<ignore_fields>N</ignore_fields>
<namingScheme/>
<shortFileFieldName/>
<extensionFieldName/>
<pathFieldName/>
<sizeFieldName/>
<hiddenFieldName/>
<lastModificationTimeFieldName/>
<uriNameFieldName/>
<rootUriNameFieldName/>
<files>
<file>
<name>${PROJECT_HOME}/output/filename_selected_fields.sql</name>
<filemask/>
<exclude_filemask/>
<file_required>N</file_required>
<include_subfolders>N</include_subfolders>
</file>
</files>
<accept_filenames>N</accept_filenames>
<accept_transform_name/>
<passing_through_fields>N</passing_through_fields>
<accept_field/>
<add_to_result_filenames>Y</add_to_result_filenames>
<error_ignored>N</error_ignored>
<file_error_field/>
<file_error_message_field/>
<skip_bad_files>N</skip_bad_files>
<bad_line_files_destination_directory/>
<bad_line_files_extension>warning</bad_line_files_extension>
<error_line_files_destination_directory/>
<error_line_files_extension>error</error_line_files_extension>
<line_number_files_destination_directory/>
<line_number_files_extension>line</line_number_files_extension>
<fields>
<field>
<name>content</name>
<position>-1</position>
<length>-1</length>
<type>String</type>
<ignore>N</ignore>
<format/>
<trim_type>none</trim_type>
<precision>-1</precision>
<currency/>
<decimal/>
<group/>
<repeat>N</repeat>
<null_string/>
<if_null_value/>
</field>
</fields>
<distribute>Y</distribute>
<copies>1</copies>
<GUI>
<xloc>176</xloc>
<yloc>288</yloc>
</GUI>
<description/>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<attributes/>
</transform>
<order>
<hop>
<from>Data grid</from>
<to>Merge join</to>
<enabled>Y</enabled>
</hop>
<hop>
<from>Text file input</from>
<to>Merge join</to>
<enabled>Y</enabled>
</hop>
<hop>
<from>Merge join</from>
<to>Filter rows</to>
<enabled>Y</enabled>
</hop>
<hop>
<from>Filter rows</from>
<to>Abort</to>
<enabled>Y</enabled>
</hop>
</order>
<notepads/>
<attributes/>
<transform_error_handling/>
</pipeline>
Loading
Loading