In FluidShell, if an SQL statement similar to the one shown below is executed:
INSERT INTO tableName(c1, c2) VALUES(1, "Test history expansion!")
this error is generated:
FluidShell: !"): event not found
This is because there is a '!' character embedded in a string quoted by double quotes which triggers history expansion in shell. We don't have a good solution to solve this problem at the moment, if user runs into this kind of problem, the workaround is to quote string constant using single quotes as:
INSERT INTO tableName(c1, c2) VALUES(1, 'Test history expansion!')
or to escape '!' using '\!' as:
INSERT INTO tableName(c1, c2) VALUES(1, "Test history expansion\!")
History expansion won't be performed if '!' is escaped as '\!' or is in a string quoted by single quotes.
|
23 KB
One more case of "!" sign causes this issue. In MongoSQL syntax sign "!" is equal to NOT. So e.g.
SELECT valuation FROM baseball where !FIELD_EXISTS(city)
Fung, we can discuss this one. I think we need to revert any history expansion on any command that is going to go into the SQL buffer
Fung, we can discuss this one. I think we need to revert any history expansion on any command that is going to go into the SQL buffer
Fixed - SVN r33005/14.0.0-beta-35
Discussed with Niels, we don't really have a good solution for this problem at the moment. SVN r33005 provides a workaround so that when history expansion failed, shell will print a warning message to stderr and continue execution. The waring message is displayed in the color defined by "File->Options->FluidShell->General->Standard Error Color". If the input is a fluid command, it is executed as a fluid command; if the input is not a fluid command, it is saved to the SQL buffer. Either way, the input is saved to the command history. The attached screenshot 'Workaround' provides some simple examples.
Fixed - SVN r33005/14.0.0-beta-35
Discussed with Niels, we don't really have a good solution for this problem at the moment. SVN r33005 provides a workaround so that when history expansion failed, shell will print a warning message to stderr and continue execution. The waring message is displayed in the color defined by "File->Options->FluidShell->General->Standard Error Color". If the input is a fluid command, it is executed as a fluid command; if the input is not a fluid command, it is saved to the SQL buffer. Either way, the input is saved to the command history. The attached screenshot 'Workaround' provides some simple examples.
Sachin & Fung, we can discuss this, but I think FluidShell should work as follows :
1. Do history expansion on original_command to get expanded_command
2. Store boolean value on whether expansion failed or not, then continue
3. Evaluate expanded_command to determine if it is a shell command or sql
4. If shell command and expansion failed, then print error and don't execute shell command
5. If shell command and expansion success, then execute shell command
6. If sql then put original_command into sql buffer
We might need to think about #6. We may need to put the expanded_command since variable substitution is also needed.
Sachin & Fung, we can discuss this, but I think FluidShell should work as follows :
1. Do history expansion on original_command to get expanded_command
2. Store boolean value on whether expansion failed or not, then continue
3. Evaluate expanded_command to determine if it is a shell command or sql
4. If shell command and expansion failed, then print error and don't execute shell command
5. If shell command and expansion success, then execute shell command
6. If sql then put original_command into sql buffer
We might need to think about #6. We may need to put the expanded_command since variable substitution is also needed.
Fung,
Can you explain what aspect of bash history expansion do we support. Specifically, do we support the following:
$ !4 service sshd restart
To execute a command that was typed 2 commands back, do the following.
$ !-2
To execute the previous command, do any one of the following:
$ !! $ !-1
You can also use keywords to execute a command from the history.
The following example will search for previous command that STARTS with the keyword “ps” and execute it. In this example, it picks up the previous command “ps -ef | grep http” and executes it.
$ !ps ps -ef | grep http
The following example will search for previous command that CONTAINS the keyword“apache” and execute it. In this example, it picks up the previous command “/usr/local/apache2/bin/apachectl restart” and executes it.
Source: http://www.thegeekstuff.com/2011/08/bash-history-expansion/
Fung,
Can you explain what aspect of bash history expansion do we support. Specifically, do we support the following:
$ !4 service sshd restart
To execute a command that was typed 2 commands back, do the following.
$ !-2
To execute the previous command, do any one of the following:
$ !! $ !-1
You can also use keywords to execute a command from the history.
The following example will search for previous command that STARTS with the keyword “ps” and execute it. In this example, it picks up the previous command “ps -ef | grep http” and executes it.
$ !ps ps -ef | grep http
The following example will search for previous command that CONTAINS the keyword“apache” and execute it. In this example, it picks up the previous command “/usr/local/apache2/bin/apachectl restart” and executes it.
Source: http://www.thegeekstuff.com/2011/08/bash-history-expansion/
See 5/1/2012 comment in issue #6693 for list of supported history events (7 events).
In addition, source comment in HistoryExpansion.java also mentions the followings:
> echo !123xxx this is to echo the event 123 plus xxx
> echo !-1X this is to echo the last executed command plus X (1 can be replaced by N)
User certainly can apply history expansion to a SQL statement if s/he likes.
See 5/1/2012 comment in issue #6693 for list of supported history events (7 events).
In addition, source comment in HistoryExpansion.java also mentions the followings:
> echo !123xxx this is to echo the event 123 plus xxx
> echo !-1X this is to echo the last executed command plus X (1 can be replaced by N)
User certainly can apply history expansion to a SQL statement if s/he likes.
I have a slightly different take on a proposed solution.
It seems that in SQL-92 and looking at MySQL, Oracle and MS SQL Server, "!" by itself is not a valid comparison operator. "!=" is valid and in SQL Server !< and !> as well. So what if EVAL_HISTORY had 3 options: true, false, fluid. In "fluid" mode, it would not expand !< != !> ?
I have a slightly different take on a proposed solution.
It seems that in SQL-92 and looking at MySQL, Oracle and MS SQL Server, "!" by itself is not a valid comparison operator. "!=" is valid and in SQL Server !< and !> as well. So what if EVAL_HISTORY had 3 options: true, false, fluid. In "fluid" mode, it would not expand !< != !> ?
> In "fluid" mode, it would not expand !< != !> ?
This might help, but the original test case described in this issue has something to with a string quoted in double quotes. None of followings would work in bash or fluid shell:
(1) INSERT INTO table(column) VALUES("!99999999")
(2) INSERT INTO table(column) VALUES("!some-text")
(3) INSERT INTO table(column) VALUES("some-test!")
!N is applied to (1) and failed.
!string is applied to (2) and (3), and failed.
> In "fluid" mode, it would not expand !< != !> ?
This might help, but the original test case described in this issue has something to with a string quoted in double quotes. None of followings would work in bash or fluid shell:
(1) INSERT INTO table(column) VALUES("!99999999")
(2) INSERT INTO table(column) VALUES("!some-text")
(3) INSERT INTO table(column) VALUES("some-test!")
!N is applied to (1) and failed.
!string is applied to (2) and (3), and failed.
with a string quoted in double quotes.
The user should be using single quotes and not double quotes. Hence, this aspect is not too much of a concern. Originally, we thought this was the only issue (double quotes vs single quotes) and had decided we could push the issue out to a future release. However, a new example was introduced: that a user could do !FIELD_EXISTS... This made us realize that in such a situation, single quotes don't appear into the equation at all.
with a string quoted in double quotes.
The user should be using single quotes and not double quotes. Hence, this aspect is not too much of a concern. Originally, we thought this was the only issue (double quotes vs single quotes) and had decided we could push the issue out to a future release. However, a new example was introduced: that a user could do !FIELD_EXISTS... This made us realize that in such a situation, single quotes don't appear into the equation at all.
> However, a new example was introduced: that a user could do !FIELD_EXISTS...
I might misunderstand this, but !FIELD_EXISTS... has something to do with "!string" history expansion, !<, !=, !> probably won't help here.
For the time being, I think you can work around "!FIELD_EXISTS..." problem by adding a space between ! and FILED_EXISTS... as "! FIELD_EXISTS...".
> However, a new example was introduced: that a user could do !FIELD_EXISTS...
I might misunderstand this, but !FIELD_EXISTS... has something to do with "!string" history expansion, !<, !=, !> probably won't help here.
For the time being, I think you can work around "!FIELD_EXISTS..." problem by adding a space between ! and FILED_EXISTS... as "! FIELD_EXISTS...".
but !FIELD_EXISTS... has something to do with "!string" history expansion
You're correct that !< != !> don't apply in this case.
So I did some more research on logical operators. Seems like SQL Server, Oracle, Sybase, DB2, PostgreSQL support the word "NOT" as a logical operator but do not support "!" (exclamation mark) as a logical operator. However, MySQL does support "!" as a logical operator. This is the only DB that I came across that supports "!". We need to verify if this assumption is correct.
In MongoSQL, we also support "!". But we can easily change this and drop support for "!" and only keep support for "NOT".
If my assumption is verified, my recommendation changes to:
-- Do nothing. If user wants "!" to be processed as a SQL Statement, then set eval_history = false
but !FIELD_EXISTS... has something to do with "!string" history expansion
You're correct that !< != !> don't apply in this case.
So I did some more research on logical operators. Seems like SQL Server, Oracle, Sybase, DB2, PostgreSQL support the word "NOT" as a logical operator but do not support "!" (exclamation mark) as a logical operator. However, MySQL does support "!" as a logical operator. This is the only DB that I came across that supports "!". We need to verify if this assumption is correct.
In MongoSQL, we also support "!". But we can easily change this and drop support for "!" and only keep support for "NOT".
If my assumption is verified, my recommendation changes to:
-- Do nothing. If user wants "!" to be processed as a SQL Statement, then set eval_history = false
I did some test using this table definition:
CREATE TABLE `issue_9392` (
`c_number` int NULL,
`c_text` varchar(25) NULL
)
on the following servers, all of SQL statements listed below are valid and can be executed successfully.
DB2 9.7 (xxx.xxx.11.57, DB2ADMIN)
SELECT * FROM DB2ADMIN."issue_9392" WHERE "c_number" != 1
SELECT * FROM DB2ADMIN."issue_9392" WHERE "c_text" != 'text'
INFORMIX 11.5 (xxx.xxx.11.38, gui_test)
SELECT * FROM informix.issue_9392 WHERE c_number != 1
SELECT * FROM informix.issue_9392 WHERE c_text != 'text'
MySQL 5.5 (xxx.xxx.11.56, test)
SELECT * FROM issue_9392 WHERE c_number != 1
SELECT * FROM issue_9392 WHERE c_text != 'text'
nCluster 4.6 (xxx.xxx.10.100, test/public)
SELECT * FROM issue_9392 WHERE c_number != 1
SELECT * FROM issue_9392 WHERE c_text != 'text'
Oracle 11g (xxx.xxx.11.51, ORCLB/test)
SELECT * FROM "test"."issue_9392" WHERE "c_number" != 1
SELECT * FROM "test"."issue_9392" WHERE "c_text" != 'text'
PostgreSQL 9.0 (xxx.xxx.11.45, AQUAFOLD/public)
SELECT * FROM "public"."issue_9392" WHERE "c_number" != 1
SELECT * FROM "public"."issue_9392" WHERE "c_text" != 'text'
SQL Server 2008 (xxx.xxx.11.57, tempdb)
SELECT * FROM [dbo].[issue_9392] WHERE c_number != 1
SELECT * FROM [dbo].[issue_9392] WHERE c_text != 'text'
However, != is not a concerned here because history expansion won't be performed if the character that follows the exclamation mark is a white space of a '=' sign.
I did some test using this table definition:
CREATE TABLE `issue_9392` (
`c_number` int NULL,
`c_text` varchar(25) NULL
)
on the following servers, all of SQL statements listed below are valid and can be executed successfully.
DB2 9.7 (xxx.xxx.11.57, DB2ADMIN)
SELECT * FROM DB2ADMIN."issue_9392" WHERE "c_number" != 1
SELECT * FROM DB2ADMIN."issue_9392" WHERE "c_text" != 'text'
INFORMIX 11.5 (xxx.xxx.11.38, gui_test)
SELECT * FROM informix.issue_9392 WHERE c_number != 1
SELECT * FROM informix.issue_9392 WHERE c_text != 'text'
MySQL 5.5 (xxx.xxx.11.56, test)
SELECT * FROM issue_9392 WHERE c_number != 1
SELECT * FROM issue_9392 WHERE c_text != 'text'
nCluster 4.6 (xxx.xxx.10.100, test/public)
SELECT * FROM issue_9392 WHERE c_number != 1
SELECT * FROM issue_9392 WHERE c_text != 'text'
Oracle 11g (xxx.xxx.11.51, ORCLB/test)
SELECT * FROM "test"."issue_9392" WHERE "c_number" != 1
SELECT * FROM "test"."issue_9392" WHERE "c_text" != 'text'
PostgreSQL 9.0 (xxx.xxx.11.45, AQUAFOLD/public)
SELECT * FROM "public"."issue_9392" WHERE "c_number" != 1
SELECT * FROM "public"."issue_9392" WHERE "c_text" != 'text'
SQL Server 2008 (xxx.xxx.11.57, tempdb)
SELECT * FROM [dbo].[issue_9392] WHERE c_number != 1
SELECT * FROM [dbo].[issue_9392] WHERE c_text != 'text'
However, != is not a concerned here because history expansion won't be performed if the character that follows the exclamation mark is a white space of a '=' sign.
On UNIX, history expansion is performed immediately after a complete line is read, before the shell breaks it into words. History expansion can be enabled/disabled using shell built-in command 'set'. FluidShell operates similarly, except it uses an on-off flag (EVAL_HISTORY shell variable) to determine whether history expansion should be performed.
The test case described in this issue is just one of two possible problems. The other problem is history expansion is unexpectedly performed against SQL statements and succeeded; when this happened, invalid data is saved to the database server.
In addition, user's input can be a compound command consists of both fluid commands and SQL statements. When history expansion is enabled, user might only want to apply history expansion to shell commands but not SQL statements. The 2-way on-off flag currently implemented by FluidShell cannot handle this.
In order to provide a reasonable solution for all of problems described above, I would suggest:
(1) FluidShell adds a new on-off flag to determine whether history expansion should be performed against SQL statements. The default setting for this flag is false. This flag won't be taken into consideration if EVAL_HISTORY is false. We can change EVAL_HISTORY to a 3-way flag (true, false, fluid) as Sachin suggested, but I would prefer adding a new flag.
(2) Change the way how FluidShell handles history expansion:
If history expansion is enabled, FluidShell needs to break user input into single commands and then apply history expansion to these commands one by one. If a single command is an SQL statement and the new flag added by (1) is off, history expansion is skipped. This likely requires Fluid to implement a new parser.
We can discuss this later on today. BTW, just reviewed history expansion related code, it seems that history expansion is always performed, EVAL_HISTORY is not implemented.
On UNIX, history expansion is performed immediately after a complete line is read, before the shell breaks it into words. History expansion can be enabled/disabled using shell built-in command 'set'. FluidShell operates similarly, except it uses an on-off flag (EVAL_HISTORY shell variable) to determine whether history expansion should be performed.
The test case described in this issue is just one of two possible problems. The other problem is history expansion is unexpectedly performed against SQL statements and succeeded; when this happened, invalid data is saved to the database server.
In addition, user's input can be a compound command consists of both fluid commands and SQL statements. When history expansion is enabled, user might only want to apply history expansion to shell commands but not SQL statements. The 2-way on-off flag currently implemented by FluidShell cannot handle this.
In order to provide a reasonable solution for all of problems described above, I would suggest:
(1) FluidShell adds a new on-off flag to determine whether history expansion should be performed against SQL statements. The default setting for this flag is false. This flag won't be taken into consideration if EVAL_HISTORY is false. We can change EVAL_HISTORY to a 3-way flag (true, false, fluid) as Sachin suggested, but I would prefer adding a new flag.
(2) Change the way how FluidShell handles history expansion:
If history expansion is enabled, FluidShell needs to break user input into single commands and then apply history expansion to these commands one by one. If a single command is an SQL statement and the new flag added by (1) is off, history expansion is skipped. This likely requires Fluid to implement a new parser.
We can discuss this later on today. BTW, just reviewed history expansion related code, it seems that history expansion is always performed, EVAL_HISTORY is not implemented.
Discussed with Sachin, we are going to revert changes made by SVN r33005 (see 6/25/13 comment), i.e. when history expansion failed, an exception is thrown and no warning will be given.
Discussed with Sachin, we are going to revert changes made by SVN r33005 (see 6/25/13 comment), i.e. when history expansion failed, an exception is thrown and no warning will be given.
Checkpoint: SVN r33083/14.0.0-beta-48
Reverted changes made by r33005 on 6/25/13.
Checkpoint: SVN r33084/14.0.0-beta-48
Performed history expansion based on the setting of the CLI_SHELL_LINE_INTERPRETER_EVAL_HISTORY shell variable.
Checkpoint: SVN r33083/14.0.0-beta-48
Reverted changes made by r33005 on 6/25/13.
Checkpoint: SVN r33084/14.0.0-beta-48
Performed history expansion based on the setting of the CLI_SHELL_LINE_INTERPRETER_EVAL_HISTORY shell variable.
Issue #9392 |
Closed |
Fixed |
Resolved |
Completion |
No due date |
Fixed Build 14.0.0-beta-35 |
No time estimate |
4 issue links |
relates to #9477
Issue #9477Fluid compound command and shell expansion/substitution |
relates to #9373
Issue #9373Error "FluidShell: !"): event not found" if use sign "!" in string |
relates to #6693
Issue #6693History expansion is not completely implemented |
depends upon #9472
Issue #9472Remove support for '!' in MongoSQL |
One more case of "!" sign causes this issue. In MongoSQL syntax sign "!" is equal to NOT. So e.g.
SELECT valuation FROM baseball where !FIELD_EXISTS(city)