Review of Linux-Fu page

By James Bailey,2014-08-10 22:31
15 views 0
Review of Linux-Fu page

    Review of Linux-Fu page

    FrequentQuestions/LinuxFu {These links seem out of place. For one, there are two Linux Fu links that just take you back to this page. For two, the other links under FAQ, do not have to do with linux. Maybe we can move this link to the main wiki page, say before or under troubleshoot?}

    1. Linux-Fu

    1. Individually tarballing a set of files

    2. Pushd/popd

    3. Command aliasing and exporting variables in .bashrc and …

    4. Persistent SCP connections (with a script)

    5. Regexp's in emacs

    Linux-Fu ?

    Got something that helps you use Linux? Know of a neat trick, or just figure something out? Let everyone know about it here.

    Individually tarballing a set of files ?

    It is often useful to tarball files before transferring them between compute clusters and local storage. If the files are large, you don't want all the files to become one tarball, but each its own tarball {Why’s that?}. A simple way to do this in bash is the following: say

    you are in a problem directory and want to tarball your combo*hdf files. Use a FOR loop to do it:

    user:~> for i in $(ls out/ch*hdf); do tar -czvf $i.tar.gz $i [&& rm

    $i]; done

    where the bracketed rm $i command, if present, will delete the original chombo file to save space.

    You can get more creative with this. Say you've tar'd files 1-10, and now want to do 20-30. Let's also say that what you're really interested in is the later files, so you'd like to tar in reverse order. The following uses seq to generate a list of numbers in reverse order,

    which are then converted to 5-digit integers via printf and export:

    user:~> for i in $(seq 30 -1 20); do

     export num=`printf %5.5i $i`;

     tar -czvf chombo$num.hdf.tar.gz out/chombo$num.hdf [&& rm



    Note the backticks on the export statement. The above can all be given on one line but is broken up here for clarity. Of course, if you want them to tar in normal order, you can simplify the above, as

    user:~> for i in {20..30}; do export ...

    If one has passwordless SSH set up, you could add a quiet scp statement for each tarball, scp chombo$num.hdf.tar.gz user@host:location &>/dev/null.

    Finally, here's a more complicated statement related to tar'ing Brick-of-value *.dat files: user:~> for i in {0..35}; do

     export prefix="W_`printf %3.3i $i`" ;

     for j in $(ls $prefix*.dat); do

     tar -czvf $j.tar.gz $j && rm -v $j;



    Push{‘}d/pop{‘}d ?

    Say you're debugging a problem module and you find that you're constantly switching between your problem directory (~/myprob) and the source directory

    (~/mycode/source). You can quickly bounce back and forth between the two with pushd and popd. pushd places your current directory and your destination directory onto a directory stack; you can subsequently alternate between the two by typing pushd

    without any arguments:

    user:~/myprob> pushd ~/mycode/

    user:~/mycode> cd source

    user:~/mycode/source> pushd

    user:~/myprob> pushd

    user:~/mycode/source> wow!

    popd removes the current directory from the directory stack and puts you into the other directory; in general it would only be used to clear the stack.

    pushd sets an environment variable $OLDPWD. This lets you greatly shorten the command

    to go back and forth from compilation and running the code. For instance, say you were editing the code in another window, and all you were doing in the current window is recompiling and running the code. You can do this all on one line like the following, which assumes you already have pushd set up:

    user:~/myprob> pushd && make mpibear && cp mpibear $OLDPWD && pushd &&

    mpirun -n 2 ./mpibear

    The use of && instead of ; will make the sequence of commands halt if one exits with an error code (e.g., if there's a problem with compilation).

Command aliasing and exporting variables in .bashrc

    and .bash_profile ?

    In your home directory, you may find a .bashrc file and a .bash_profile

    file. .bash_profile executes when you open a terminal that requires a login (such as SSH), whereas .bashrc opens for a "non-login interactive shell", such as you might open in X with GNOME or KDE. In practice, .bash_profile will usually contain a line

    invoking .bashrc, so there's no effective difference.

    There are several tricks with .bashrc to make your life easier. The first is the alias

    command. alias maps a command word to a more detailed expression that the shell executes. By adding commands of the form alias <command>="<bash shell

    expression>" to your .bashrc file you can map frequently-used expressions to shorter commands. For instance, you can make sure that you always enable X11 forwarding in SSH with the line

    alias ssh="ssh -Y"

    Or you can specify the build of VisIt in /opt/visit/bin to execute on the command

    visit by entering

    alias visit="/opt/visit/bin/visit"

    Any expression you might type into a terminal can be aliased in this fashion. To apply the changes to your .bashrc file without logging out, type

    source ~/.bashrc

    when you're done. This reads and re-executes the commands in .bashrc.

    Another useful trick in .bashrc is exporting variables to the environment. By including lines of the form export VARIABLE_NAME=<variable_value>, we make the variable

    $VARIABLE_NAME accessible within the command-line environment.

    This is especially useful when applied to the $PATH and $LD_LIBRARY_PATH variables.

    These pre-existing environment variables contain the paths Linux searches to look for executables and shared library objects, respectively. Examples of adding new paths to these variables are below:

    export PATH=$PATH:/usr/local/bin

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hdf5/lib

    Note the use of $PATH and $LD_LIBRARY_PATH in these variables. This effectively

    concatenates the new path to any existing list of paths in the variable. Remember that Linux searches these paths in order, so if you want one path to appear before the rest you should use export <new_path>:$PATH instead.

    source ~/.bashrc into the command line when you are done Again, remember to type

    editing .bashrc to implement the changes immediately.

    Persistent SCP connections (with a script) ?

    To see this page, please go here.

    Regexp's in emacs ?

    Regular expressions are not intuitive at the best of times. Trying to figure out what's going on in emacs only makes the problem worse.

    So, here's an example: I have several functions I want to rename, all of the form C_name1(a,b,c), C_name2(a,b,c), etc., to add an extra underscore, i.e.,

    C_name1_(a,b,c). I can do this in emacs with the following:

    (M-X) replace-regexp

    (1st prompt) \(C_\w*\)(

    (2nd prompt) \1_(

    In the first prompt (the to-be-replaced string), you can specify substrings using the \( and \) characters to group them. So I've wrapped the function name in such a group. The \w character will find any word character (no whitespace), and the * finds any number of them. It stops when it reaches the (, which is the next non-wildcard character in the to-be-replaced string.

    Emacs allows you to have multiple groups in the to-be-replaced string, which you can reference in your replacement string. They are referenced left-to-right by \1, \2, etc. So here, I indicate that in the new string I want the first group (e.g. "C_name1") to come first, followed by a _(.

    Conversely, if you want to pre/append an entire string (say, change all foot's to football's), you could use the \& character which represents the entire to-be-replaced string.

Report this document

For any questions or suggestions please email