Ruby fails in calling a bash script

(Thursday, May 12, 2011, at 03:29 PM)

so today, I found myself working on some code that would eventually execute a monstrous query in bash:

comm -12 <(cat one.txt) <(comm -12 <(cat two.txt) <(comm -12 <(cat three.txt) <(cat four.txt)

Basically, this is a comparison of four files, where the result would be the similar items across each of the files. Now, unassuming me thought that I could just call the command via ruby:


`comm -12 <(cat one.txt) <(comm -12 <(cat two.txt) <(comm -12 <(cat three.txt) <(cat four.txt)))`

But that results in an error:


sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `comm -12 <(cat one.txt) <(comm -12 <(cat two.txt) <(comm -12 <(cat three.txt) <(cat four.txt)))'

So, then I thought maybe you could write a file, and then execute that sh file instead:


def all_combinations(array)
  permutations = []
  array.length.downto(2) do |length|
    array.permutation(length).each do |perm|
      permutations << perm.sort if !permutations.include?(perm.sort)
    end
  end
  return permutations
end

def comm_partial(files, index=0, text="")
  if files.length > 2
    file = files[index]
    if files.last==file
      text+="<(cat #{file})"
    elsif files.first==file
      text+="<(cat #{files[index]}) #{comm_partial(files, index+1, text)})"
    else
      text+="<(comm -12 <(cat #{files[index]}) #{comm_partial(files, index+1, text)})"
    end
    if files.first==file
      return "comm -12 "+text.chop
    else
      return text
    end
  else
    return "comm -12 #{files[0]} #{files[1]}"
  end
end
files = ['one','two','three','four']
all_combos = all_combinations(files)
shell_script = File.open("script.sh", "w")
shell_script.write("#!/bin/sh")
all_combos.each do |combo|
  files = combo.collect{|file| "#{file}.txt"}
  function = comm_partial(files)+">>#{combo.join("_")}.txt\n"
  shell_script.write(function)
end
shell_script.close
`chmod +x script.sh`
runfile = File.open("run.rb", "w")
runfile.write("`./script.sh`")
runfile.close

and then just ruby run.rb in terminal to get my results. BUT NO:


./script.sh: line 2: syntax error near unexpected token `('
./script.sh: line 2: `comm -12 <(cat four.txt) <(comm -12 <(cat one.txt) <(comm -12 <(cat three.txt) <(cat two.txt)))>>four_one_three_two.txt'

AGAIN I am thwarted. SO, I go to irc and post up this gist file, and zenspider explained it like this:

[3:21pm] zenspider: dgaffney: ruby will shell out to /bin/sh if it has special shell chars in it
[3:21pm] zenspider: /bin/sh != /bin/bash (even tho it is)
[3:21pm] dgaffney: zenspider: can you force backtick/system/exec/popen to use bash instead?
[3:22pm] zenspider: dgaffney: honestly… I dunno. you could always wrap it up in your own bash -c ‘…’, but that seems sketchy to
[3:22pm] zenspider: too
[3:23pm] dgaffney: and what is the special shell char in question here?
[3:23pm] zenspider: I know that this is probably just a simple example of your real problem, but my guess is you can avoid all those cats and therefore avoid nearly all the clever bashisms
[3:24pm] dgaffney: zenspider: Yeah, i would just use unions against the ints
[3:24pm] zenspider: dgaffney: anything like <>&| etc
[3:24pm] dgaffney: but the real files are millions of ints long
[3:24pm] dgaffney: so unions are going to not work long term
[3:25pm] dgaffney: bash -c seems to do it
[3:25pm] zenspider: dgaffney: no… `prog <(cat f1) <(cat f2)` should be the same as `prog f1 f2`
[3:25pm] zenspider: granted, you’re doing it against 3, but still
[3:26pm] zenspider: you’re using up extra needless resources regardless
[3:26pm] dgaffney: yeah
[3:26pm] zenspider: dgaffney: in your example, I think you can do it w/o any bashisms
[3:27pm] dgaffney: you can’t really get something to do the union without temping each individual comparison
[3:27pm] zenspider: dgaffney: comm -12 one.txt two.txt | comm -12 – three.txt
[3:27pm] dgaffney: hmmmmmmmmmmm
[3:28pm] zenspider: dgaffney: you should be able to chain that as far as you need it
[3:28pm] dgaffney: and without my massive recursive function
[3:28pm] dgaffney: to generate that bash command
[3:29pm] dgaffney: well, alright
[3:29pm] dgaffney: fuckit
[3:29pm] dgaffney: you answered an age-old question though, so thats awesome
[3:29pm] dgaffney: steveklabnik: turns out there was a reasonable explanation behind the problem
[3:29pm] steveklabnik: dgaffney: yeah, i saw

So, to end the conversation, The reasonable response to this problem can go one of a few ways:

1. Instead of #!/bin/sh at the beginning of the shell script, refer it to #!/bin/bash instead
2. Try your very, very, very hardest to avoid characters like <>&| etc.

Back