Shell script vs C performance

I suspect, based on your description, that you're spawning off new processes in your shell script. If that's the case, then that's where your time is going. It takes a lot of OS resource to fork/exec a new process.

As donitor and Dietrich sugested, I did a little research about the AWK language and, again, as they said, it was a total success. here is a little example of the AWK program:

#!/bin/awk -f{    option=substr($0, 5, 9);    if (option=="SOMETHING"){        type=substr($0, 80, 1)        if (type=="A"){            type="01";        }else if (type=="B"){            type="02";        }else if (type=="C"){            type="03";        }        print substr($0, 7, 3) substr($0, 49, 8) substr($0, 86, 8) type\        substr($0, 568, 30) >> ARGV[2]    }}

And it works like a charm. It takes barely 1 minute to process a 500mb file

What's wrong with the C program? Is it broken? Too hard to maintain? Too inflexible? You are more of a Shell than a C expert?

If it ain't broke, don't fix it.

A look at Perl might be an option, too. Easier than C to modify and still speedy I/O; and it's much harder to create useless forks in Perl than in the shell.

If you told us exactly what the C program does, maybe there's a simple and faster-than-light solution with sed, grep, awk or other gizmos in the Unix tool box. In other words, tell us what you actually want to achieve, don't ask us to solve some random problem you ran into while pursuing what you think is a step towards your actual goal.

Alright, one problem with your shell script is the repeated open in echo "$out" >> "outFileName". Use this instead:

while read line; do    echo "${line:10:16}.${line:45:2}${line:106:61}${line:189:3}${line:215:15}..." done < "$fileName" > "$outFileName"

As an alternative, simply use the cut utility (but note that it doesn't insert the dot after the first part):

cut -c 10-26,45-46,106-166 "$fileName" > "$outFileName"

You get the idea?