Using sed to match regexp patterns in a file with a fixed string (not the other way around)

2 years ago by silas to c/shell

Hi there!

Usually, sed can be used in different ways, but most of the time we use it to match lines in a file against a fixed regexp. Some examples:

This replaces ocurrences of regexp for "foo":

sed 's/regexp/foo/g' < myfile

This prints all lines that have "foo", but will change the first "o" in the line for an "a":

sed -n '/foo/s/o/a/p' < myfile

and so on...

But I tried to do a different thing, with no success: can I pass to sed a file with a bunch of regular expressions and test them against a fixed string? I tried to play with pattern space, hold space, with no success. It just seems impossible to use them (which would be the closest to "variables") in search commands.

I know sed is Turing complete, but using it that way would maybe require to implement a regexp engine from scratch?

Thanks!

silas 4 points 2 years ago

Yeah, I ended up using awk, which solved my problem perfectly. I was just curious if I could do that with sed, but it seems too complicated. Thank very much, guys!

path: 0 14545388, hotness: undefined, score: 4, children: 1
zero_spelled_with_an_ecks 1 point 2 years ago

Good call. I have no doubt sed could do it but I do doubt it in would be worth the time to figure it out vs just adding awk in there.

path: 0 14545388 14545431, hotness: undefined, score: 1, children: 0
notabot 1 point 2 years ago

What result do you need? Do you just nedd to know if one of the regexps matched the string, or do you need to know which one, or is it something else?

path: 0 14543209, hotness: undefined, score: 1, children: 0
ccryx 1 point 2 years ago

I know you asked about sed, but in case grep is fine, too:

echo theString | grep -f <file with patterns>

Not sure if there's an output which pattern matched.

path: 0 14544433, hotness: undefined, score: 1, children: 0
FizzyOrange -3 points 2 years ago

IMO if you're doing something that complex you shouldn't be using sed, but yeah you can probably do this something like:

while read REGEX; do
  sed "$REGEX" << EOF
your test string
EOF
done <list_of_regexes.txt

I strongly recommend you don't do that though. It will be absolutely full of quoting bugs. Instead write a script in a proper language to do it. I recommend Deno, or maybe even Rust.

If you use Rust you can also use RegexSet which will be much faster (if you just want to find matches anyway). Here's what ChatGPT made me. Not tested but it looks vaguely right.

use regex::RegexSet;
use std::fs::File;
use std::io::{self, BufRead};
use std::path::Path;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Define the fixed input string
    let input_string = "This is a test string for regex matching.";

    // Path to the file containing the regexes
    let regex_file_path = "regexes.txt";

    // Read regexes from the file
    let regexes = read_lines(regex_file_path)?
        .filter_map(Result::ok) // Filter out errors
        .collect::<Vec<String>>();

    // Create a RegexSet from the regexes
    let regex_set = RegexSet::new(&regexes)?;

    // Find the regexes that match the input string
    let matches: Vec<_> = regex_set.matches(input_string).into_iter().collect();

    // Print the matches
    if matches.is_empty() {
        println!("No regexes matched the input string.");
    } else {
        println!("Regexes that matched the input string:");
        for index in matches {
            println!(" - {}", regexes[index]);
        }
    }

    Ok(())
}

// Helper function to read lines from a file
fn read_lines<P>(filename: P) -> io::Result<io::Lines<io::BufReader<File>>>
where
    P: AsRef<Path>,
{
    let file = File::open(filename)?;
    Ok(io::BufReader::new(file).lines())
}
path: 0 14541574, hotness: undefined, score: -3, children: 0
shell
shell

@programming.dev

login for more options
1615
50
1

From Ash, Bash and Csh to Xonsh, Ysh and Zsh; all shell languages are welcome here!

Rules:
  1. Follow Lemmy rules!
  2. Posts must relate to shell scripting. (See bottom of sidebar for more information.)
  3. Only make helpful replies to questions. This is not the place for low effort joke answers.
  4. No discussion about piracy or hacking.
  5. If you find a solution to your problem by other means, please take your time to write down the steps you used to solve your problem in the original post. You can potentially help others having the same problem!
  6. These rules will change as the community grows.

Keep posts about shell scripting! Here are some guidelines to help:

  • Allowed: Release Announcement of a command-line program designed for scripted use (e.g. bash, bats, awk, jq, coreutils/moreutils)
  • Allowed: Tutorials on using shell languages or supplementary tools designed for scripted use
  • Allowed: Code review/help requests for shell languages or supplementary tools designed for scripted use
  • NOT Allowed: Announcement of a CLI or TUI program that is not designed for scripted use (Yes, your fancy Zsh plugin which pretty-prints the date and time using only builtins is very cool, but unless you actually want to discuss the code itself, please check out !commandline instead!)
  • NOT Allowed: Domain-specific tutorials that do not involve shell scripting as a core component (e.g. docker-compose, ansible, nix). If you really love one of these, I'm sure there's a community out there ready to talk about them!
  • NOT Allowed: Code review requests for non-shell programming languages and configuration languages (e.g. Python, Yaml)

In general, if your submission text is primarily shell code, then it is welcome here!

go to feed...