Ruby fails in calling a bash script
(Thursday, May 12, 2011, at 03:29 PM)
so today, I found myself working on some code that would eventually execute a monstrous query in bash:
comm -12 <(cat one.txt) <(comm -12 <(cat two.txt) <(comm -12 <(cat three.txt) <(cat four.txt)
Basically, this is a comparison of four files, where the result would be the similar items across each of the files. Now, unassuming me thought that I could just call the command via ruby:
`comm -12 <(cat one.txt) <(comm -12 <(cat two.txt) <(comm -12 <(cat three.txt) <(cat four.txt)))`
But that results in an error:
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `comm -12 <(cat one.txt) <(comm -12 <(cat two.txt) <(comm -12 <(cat three.txt) <(cat four.txt)))'
So, then I thought maybe you could write a file, and then execute that sh file instead:
def all_combinations(array)
permutations = []
array.length.downto(2) do |length|
array.permutation(length).each do |perm|
permutations << perm.sort if !permutations.include?(perm.sort)
end
end
return permutations
end
def comm_partial(files, index=0, text="")
if files.length > 2
file = files[index]
if files.last==file
text+="<(cat #{file})"
elsif files.first==file
text+="<(cat #{files[index]}) #{comm_partial(files, index+1, text)})"
else
text+="<(comm -12 <(cat #{files[index]}) #{comm_partial(files, index+1, text)})"
end
if files.first==file
return "comm -12 "+text.chop
else
return text
end
else
return "comm -12 #{files[0]} #{files[1]}"
end
end
files = ['one','two','three','four']
all_combos = all_combinations(files)
shell_script = File.open("script.sh", "w")
shell_script.write("#!/bin/sh")
all_combos.each do |combo|
files = combo.collect{|file| "#{file}.txt"}
function = comm_partial(files)+">>#{combo.join("_")}.txt\n"
shell_script.write(function)
end
shell_script.close
`chmod +x script.sh`
runfile = File.open("run.rb", "w")
runfile.write("`./script.sh`")
runfile.close
and then just ruby run.rb in terminal to get my results. BUT NO:
./script.sh: line 2: syntax error near unexpected token `('
./script.sh: line 2: `comm -12 <(cat four.txt) <(comm -12 <(cat one.txt) <(comm -12 <(cat three.txt) <(cat two.txt)))>>four_one_three_two.txt'
AGAIN I am thwarted. SO, I go to irc and post up this gist file, and zenspider explained it like this:
[3:21pm] zenspider: dgaffney: ruby will shell out to /bin/sh if it has special shell chars in it
[3:21pm] zenspider: /bin/sh != /bin/bash (even tho it is)
[3:21pm] dgaffney: zenspider: can you force backtick/system/exec/popen to use bash instead?
[3:22pm] zenspider: dgaffney: honestly… I dunno. you could always wrap it up in your own bash -c ‘…’, but that seems sketchy to
[3:22pm] zenspider: too
[3:23pm] dgaffney: and what is the special shell char in question here?
[3:23pm] zenspider: I know that this is probably just a simple example of your real problem, but my guess is you can avoid all those cats and therefore avoid nearly all the clever bashisms
[3:24pm] dgaffney: zenspider: Yeah, i would just use unions against the ints
[3:24pm] zenspider: dgaffney: anything like <>&| etc
[3:24pm] dgaffney: but the real files are millions of ints long
[3:24pm] dgaffney: so unions are going to not work long term
[3:25pm] dgaffney: bash -c seems to do it
[3:25pm] zenspider: dgaffney: no… `prog <(cat f1) <(cat f2)` should be the same as `prog f1 f2`
[3:25pm] zenspider: granted, you’re doing it against 3, but still
[3:26pm] zenspider: you’re using up extra needless resources regardless
[3:26pm] dgaffney: yeah
[3:26pm] zenspider: dgaffney: in your example, I think you can do it w/o any bashisms
[3:27pm] dgaffney: you can’t really get something to do the union without temping each individual comparison
[3:27pm] zenspider: dgaffney: comm -12 one.txt two.txt | comm -12 – three.txt
[3:27pm] dgaffney: hmmmmmmmmmmm
[3:28pm] zenspider: dgaffney: you should be able to chain that as far as you need it
[3:28pm] dgaffney: and without my massive recursive function
[3:28pm] dgaffney: to generate that bash command
[3:29pm] dgaffney: well, alright
[3:29pm] dgaffney: fuckit
[3:29pm] dgaffney: you answered an age-old question though, so thats awesome
[3:29pm] dgaffney: steveklabnik: turns out there was a reasonable explanation behind the problem
[3:29pm] steveklabnik: dgaffney: yeah, i saw
So, to end the conversation, The reasonable response to this problem can go one of a few ways:
1. Instead of #!/bin/sh at the beginning of the shell script, refer it to #!/bin/bash instead
2. Try your very, very, very hardest to avoid characters like <>&| etc.