How To Run Shell Command And Get Output In Python

In the early python version, the os module’s system or popen function is usually used to execute operating system commands. However, recently python has gradually abandoned these functions officially. It recommends using the built-in subprocess module to execute operating system-related commands. This article will tell you how to use the subprocess module and os module’s system, popen function to run a shell command.

1. Run Shell Command Use subprocess Module.

  1. First, we should import the subprocess module.
    >>> import subprocess
  2. Then we should create a child process object using the subprocess module’s Popen method and assign it to a variable.
    # the child process will print it's standard output to a pipe line, the pipe line connect the child process and it's parent process.
    
    >>> child = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  3. Then we can get the subprocess execution result output text using the child process’s stdout read() method.
    >>> print(child.stdout.read())
  4. Below is another example that runs a command java -version to get the currently installed java version.
    >>> import subprocess
    
    >>> child = subprocess.Popen(['java','-version'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    
    >>> print(child.stdout.read())
    b'openjdk version "1.8.0_212"\nOpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b03)\nOpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b03, mixed mode)\n'

2. Run Shell Command Use os Module system And popen Function.

2.1 os.system(command).

  1. Run operating system commands and display the results directly on the standard output device( ie: screen console).
  2. But the function’s return value is 0 or -1, and the data displayed on the screen cannot be obtained in the source code.
  3. The command parameter is the command string to be executed.
    >>> import os
    >>> ret = os.system('java -version')
    openjdk version "1.8.0_212"
    OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_212-b03)
    OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.212-b03, mixed mode)
    >>> ret
    0

2.2 os.popen(command, [mode, [bufsize]]).

  1. Start a child process to execute the command specified by the command parameter, and establish a pipeline between the parent process and the child process for communication between the parent and child processes.
  2. This method returns a file object, which can be read or written, depending on the parameter mode‘s value.
  3. If the mode parameter’s value is ‘r’, then the file is read-only. If the mode parameter’s value is ‘w’, then the file is write-only and it will throw an error when you want to get the file object’s content using it’s read() method.
  4. In short, popen method can run operating system commands and can return the result of the command through the result file object’s read() method.
  5. Below is an example.
    >>> import os 
    
    # invoke os.popen method to run a shell command.
    >>> ret = os.popen('ifconfig')
    
    # get the shell command execution result through the returned file object's read() method.
    >>> ret.read()
    'lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384\n\toptions=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>\n\tinet 127.0.0.1 netmask 0xff000000 \n\tinet6 ::1 prefixlen 128 \n\tinet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 \n\tnd6 options=201<PERFORMNUD,DAD>\ngif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280\nstf0: flags=0<> mtu 1280\nen0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500\n\toptions=50b<RXCSUM,TXCSUM,VLAN_HWTAGGING,AV,CHANNEL_IO>\n\tether 10:dd:b1:9b:b0:b9 \n\tnd6 options=201<PERFORMNUD,DAD>\n\tmedia: autoselect (none)\n\tstatus: inactive\nen1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500\n\toptions=400<CHANNEL_IO>\n\tether 4c:8d:79:e1:bc:e2 \n\tinet6 fe80::82a:736f:f606:31f0%en1 prefixlen 64 secured scopeid 0x5 \n\tinet 192.168.31.31 netmask 0xffffff00 broadcast 192.168.31.255\n\tnd6 options=201<PERFORMNUD,DAD>\n\tmedia: autoselect\n\tstatus: active\nen3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500\n\toptions=460<TSO4,TSO6,CHANNEL_IO>\n\tether 82:0a:57:17:dd:80 \n\tmedia: autoselect <full-duplex>\n\tstatus: inactive\nfw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 4078\n\tlladdr 10:dd:b1:ff:fe:5c:5f:76 \n\tnd6 options=201<PERFORMNUD,DAD>\n\tmedia: autoselect <full-duplex>\n\tstatus: inactive\nbridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500\n\toptions=63<RXCSUM,TXCSUM,TSO4,TSO6>\n\tether 82:0a:57:17:dd:80 \n\tConfiguration:\n\t\tid 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0\n\t\tmaxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200\n\t\troot id 0:0:0:0:0:0 priority 0 ifcost 0 port 0\n\t\tipfilter disabled flags 0x0\n\tmember: en3 flags=3<LEARNING,DISCOVER>\n\t        ifmaxaddr 0 port 6 priority 0 path cost 0\n\tnd6 options=201<PERFORMNUD,DAD>\n\tmedia: <unknown type>\n\tstatus: inactive\np2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304\n\toptions=400<CHANNEL_IO>\n\tether 0e:8d:79:e1:bc:e2 \n\tmedia: autoselect\n\tstatus: inactive\nawdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484\n\toptions=400<CHANNEL_IO>\n\tether 02:c2:69:6d:b3:5e \n\tinet6 fe80::c2:69ff:fe6d:b35e%awdl0 prefixlen 64 scopeid 0xa \n\tnd6 options=201<PERFORMNUD,DAD>\n\tmedia: autoselect\n\tstatus: active\nutun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380\n\tinet6 fe80::a4be:4aed:be5b:253%utun0 prefixlen 64 scopeid 0xb \n\tnd6 options=201<PERFORMNUD,DAD>\nutun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000\n\tinet6 fe80::cfcf:1568:4d94:3b0a%utun1 prefixlen 64 scopeid 0xc \n\tnd6 options=201<PERFORMNUD,DAD>\n'

3. Other Python os, shutil Module Functions Example.

  1. os.listdir(dirname): This function returns a list of strings containing all file names under the directory, but excluding “.” and “..”.
    >>> import os
    >>>
    >>> os.listdir('.\\')
    ['.anaconda', '.cache', '.conda', '.condarc', '.config', '.continuum', '.eclipse', '.FAILED_FILE_CACHED_DIR', '.gitconfig', '.lemminx', '.m2', '.matplotlib', '.p2', '.VirtualBox']
  2. os.getcwd(): Returns the current working directory path.
    >>> import os
    >>>
    >>> os.getcwd()
    'C:\\Users\\zhaosong'
  3. os.chdir(dirname): Switch the current working directory to dirname.
    >>>import os
    >>>
    >>>os.getcwd()
    'D:\\'
    >>>
    >>>os.chdir('c:\\')
    >>>
    >>>os.getcwd()
    “c:\\”
  4. os.chroot(dirname): Make dirname the root directory of the process. It is similar to the chroot command under * Nix
  5. os.chmod(path,mode): Change the permission bit of path. Mode can be a combination of the following values (using or).
    os.S_ISUID, os.S_ISGID, os.S_ENFMT, os.S_ISVTX, os.S_IREAD, os.S_IWRITE, os.S_IEXEC, os.S_IRWXU, os.S_IRUSR, os.S_IWUSR, os.S_IXUSR
    , os.S_IRWXG, os.S_IRGRP, os.S_IWGRP, os.S_IXGRP, os.S_IRWXO, os.S_IROTH, os.S_IWOTH, os.S_IXOTH
    
    R stands for read, W for write, and X for execute permission. USR represents the user, GRP represents the group, OTH represents others.
  6. os.link(src, dest): Create hard connection.
  7. os.chown(path, uid, gid): Change the ownership of the file. When uid and gid are – 1, the original owner will not be changed.
  8. os.mkdir(path,[mode]): Create a directory. See os.chmod() for the meaning of mode. The mode default value is 0777.
  9. os.makedirs(path,[mode]): Similar to os.mkdir(), but the nonexistent parent directory will be created first.
  10. os.readlink(path): Returns the path to which the symbolic link points.
  11. os.remove(path): Delete file, cannot be used to delete a directory.
  12. os.rmdir(path): Delete folder cannot be used to delete files.
  13. os.symlink(src, dest): Create symbolic links.
  14. os.getenv(), os.putenv(): Used to read and set environment variables.
  15. os.system(‘md a’) : Create a directory directly under the current directory.
  16. os.name: Indicates the platform you are using. For example, it is ‘nt‘ for Windows and ‘posix‘ for Linux/Unix users.
  17. os.linesep: This string gives the line terminator used by the current platform. For example, Windows OS uses ‘ \r\n ‘, Linux OS uses ‘\n‘ and MAC OS uses ‘\r‘.
  18. shutil.copy(src, dest): The parameter src refers to the name of the source file, and the parameter dest refers to the name of the target file or target directory. If dest is a directory name, a file with the same name will be created in that directory.
  19. shutil.copy2(src, dest): Similar to the shutil.copy() function is shutil.copy2 (src, dest), but function copy2() also copies the last access time and the last update time.
  20. shutil.copytree(src, dest[, symlinks]): This function can copy the directory. The parameter symlinks is a Boolean value. If it is true, a symbolic link will be created.
  21. shutil.move(src, dest): This function can move or rename files and directories. If src and dest are on the same file system, shutil.move() simply changes the name. If src and dest are on different file systems, shutil.move() will first copy src to dest and then delete the src file.
  22. shutil.rmtree(path[,ignore_errors[,onerror]]): remove folders.

4. Python System Environment Variable.

  1. The python environment variable is stored in the os.environ dictionary. It can be modified by using the ordinary dictionary method. It will be inherited automatically when starting other programs with the system.
  2. The value of a python environment variable can only be a string. Unlike the shell, python does not have the concept of exporting environment variables.
  3. The below example will add a python environment variable.
    os.environ[”hello”]=”haha”

5. The os.path Module.

  1. This module contains many functions for pathname processing. In the shell, pathname processing seems not very important, but it is often used in Python. The two most commonly used are separating and merging directory names and file names.
  2. os.path.split(path) -> (dirname, basename): This function will separate a path into two parts.
    >>> import os
    >>>
    >>> os.path.split("/foo/bar.dat")
    ('/foo', 'bar.dat')
  3. os.path.join(dirname, basename): This function will combine the directory name and file name into a complete pathname. This function is just the opposite of os.path.split().
    >>> import os
    >>>
    >>> os.path.join("/foo","bar.dat")
    '/foo\\bar.dat'
  4. os.path.abspath(path): Convert path to absolute path.
  5. os.path.expanduser(path): Convert “~” and “~ user” contained in path into user directory.
  6. os.path.expandvars(path): Replace “$name” and “${name}” contained in the path to the value of the environment variable accordingly. For example, if the environment variable is Hello = haha, then os.path.expandvars (“$Hello/ ABC”) will return “haha/ ABC“.
  7. os.path.normpath(path): Remove the “.” and “..” contained in the path.
  8. os.path.splitext(path): Separate path into base name and extension name.
    >>> import os
    >>>
    >>> os.path.splitext("/foo /bar.tar.bz2")
    ('/foo /bar.tar', '.bz2')
  9. os.path.exists(path): Determine whether the file or directory exists.
  10. os.path.isfile(): Determine whether the path points to an ordinary file rather than a directory.
  11. os.path.isdir(path): Determine whether the path points to a directory rather than an ordinary file.
  12. os.path.islink(path): Determine whether the path points to a symbolic link.
  13. os.path.ismount(path): Judge whether the path points to a mount point.
  14. os.path.getatime(path): Returns the last access time of the file or directory pointed to by path.
  15. os.path.getmtime(path): Returns the last modification time of the file or directory pointed to by path.
  16. os.path.getctime(path): Returns the creation time of the file pointed to by path.
  17. os.path.getsize(path): Returns the size of the file pointed to by path.

6. Run Shell Command In Python Examples.

6.1 The following is a simple script case for copying files.

  1. There is a directory of tens of thousands of files.
  2. I want to copy these files to other directories, but I do not want to copy the directory itself directly.
  3. I tried “cp src/ * dest/” and it reported an error that the command line was too long.
  4. So I write the below python source code to implement the task.
    import sys,os.path,shutil
    
    # sys.argv[0] is the python program name.
    # sys.argv[1] is the source directory.
    # sys.argv[2] is the target directory.
    
    # loop in the source directory. 
    for f in os.listdir(sys.argv[1]):
          
          # get the source file full path value.
          src_file_path = os.path.join(sys.argv[1],f)
          # copy the source file to the target directory
          shutil.copy(src_file_path,sys.argv[2])

6.2 The following example renames all files in a folder to 10001 ~ 10999.

  1. Below is the example source code.
    import os.path,sys
    
    # get the files source folder.
    dirname=sys.argv[1]
    
    i = 10001
    
    # loop all the files in the directory.
    for f in os.listdir(dirname):
    
        # get the source file.
        src=os.path.join(dirname,f)
    
        # check whether the file is a directory or not.
        if os.path.isdir(src):
            
            # if the file is a directory then continue. 
            continue
        # otherwise rename the file name.
        os.rename(src,str(i))
    
        i += 1

7. File Redirection.

  1. There is an existing py file new1.py.
  2. When you enter the command new1 > new.txt on the command line to output the running result of new1 to the file new.txt, this is called stream redirection