Skip to content

xebialabs/overthere

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1,242 Commits
examples
examples
 
 
gradle/wrapper
gradle/wrapper
 
 
src
src
 
 
.gitignore
.gitignore
 
 
HEADER
HEADER
 
 
Jenkinsfile
Jenkinsfile
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
build.gradle
build.gradle
 
 
gradlew
gradlew
 
 
gradlew.bat
gradlew.bat
 
 
settings.gradle
settings.gradle
 
 

Repository files navigation

Table of Contents

Introduction

Overthere is a Java library to manipulate files and execute processes on remote hosts, i.e. do stuff "over there". Overthere was originally developed for and is used in the XebiaLabs deployment automation product Deployit as a way to perform tasks on remote hosts, e.g. copy configuration files, install EAR files or restart web servers. Another way of looking at it is to say that Overthere gives you java.io.File and java.lang.Process as they should've been: as interfaces, created by a factory and extensible through an SPI mechanism.

Overthere is available under the GPLv2 with XebiaLabs FLOSS License Exception.

P.S.: Check the Overthere Ohloh page for some interesting code analysis statistics. If you use Overthere, don't forget to tell Ohloh! And while you're at it, you might want to vote for Overthere on the Overthere Freecode page too! ;-)

Getting Overthere

To get Overthere, you have two options:

  1. Add a dependency to Overthere to your project.
  2. Build Overthere yourself.

And, if you want, you can also run the Overthere examples used in the Overthere presentation mentioned above.

Binary releases of Overthere are not provided here, but you can download it straight from the Maven Central repository if you want to.

Depending on Overthere

  1. If your project is built with Maven, add the following dependency to the pom.xml:

     <dependency>
 	<groupId>com.xebialabs.overthere</groupId>
 	<artifactId>overthere</artifactId>
 	<version>5.3.2</version>
 </dependency>

  2. If your project is built using another build tool that uses the Maven Central repository, translate these dependencies into the format used by your build tool.

Building Overthere

  1. Clone the Overthere repository.
  2. On unix run the command ./gradlew clean build, on windows run gradlew clean build

Running the examples

  1. Install Maven 2.2.1 or up.
  2. Clone the Overthere repository.
  3. Go into the examples directory and run the command mvn eclipse:eclipse.
  4. Import the examples project into Eclipse.
  5. Change the login details in the example classes (address, username and password) and run them!

Programming Overthere

To program Overthere, browse the source code, check the examples and browse the Overthere Javadoc.

For a more thorough introduction to Overthere, check the presentation on Overthere that I gave for J-Fall 2011, a Java conference in the Netherlands (in English).

Configuring Overthere

The protocols that Overthere uses to connect to remote hosts, such as SSH, CIFS, Telnet and WinRM, are existing protocols for which support is built into many platforms. As such you will not need to install any custom software on the remote hosts. Nevertheless in some cases the remote hosts have to be configured to correctly work with Overthere. Also, Overthere has a number of configuration features that allow you tweak the way it interfaces with the remote hosts.

Protocols

Overthere supports a number of protocols to connect to remote hosts:

  • local - a connection to the local host. This is a wrapper around java.io.File and java.lang.Process.
  • ssh - a connection using the SSH protocol, to a Unix host, to a z/OS host, or to a Windows host running either OpenSSH on Cygwin (i.e. COPSSH) or WinSSHD or microsoft OpenSSH.
  • smb - a connection using the prevalent SMB protocol for file manipulation and, depending on the settings, using either WinRM or Telnet for process execution. This protocol is only supported for Windows hosts.
  • cifs - a connection using public variant of the original Server Message Block (SMB) protocol developed by Microsoft known as CIFS protocol, for file manipulation and, depending on the settings, using either WinRM or Telnet for process execution. This protocol is only supported for Windows hosts, CIFS is widely regarded as an obsolete protocol and users are encouraged to prefer a SMB protocol over a CIFS. Support for CIFS is deprecated and will be removed from subsequent releases.
  • ssh-jumpstation - a special protocol type that can only be used as a jumpstation protocol, which allows a connection to be created over an SSH jumpstation.
  • proxy - a special protocol type that can only be used as a jumpstation protocol, which allows a connection to be created over an HTTP proxy.

Common connection options

Apart from selecting a protocol to use, you will also need to supply a number of connection options when creating a connection. Common connection options are:

protocol This option is only used when it is present in the connection options of a tunnel (jumpstation) connection. It indicates which protocol is used to tunnel the other connection over. This property can be set to ssh or proxy. If this option is omitted in the jumpstation options, ssh is assumed.
os The operating system of the remote host. This property can be set to UNIX, WINDOWS, and ZOS and is used to determine how to encode paths and commands and to determine the default temporary directory path. This property is required for all protocols, except for the local protocol where it is automatically determined.
address The address of the remote host, either an IP address or a DNS name. This property is required for all protocols, except for the local protocol.
port The port to use when connecting to the remote host. The interpretation and the default value for this connection option depend on the protocol that is used.
username The username to use when connecting to the remote host. This property is required for all protocols, except for the local protocol.
password The password to use. This property is required for all protocols, except for the local protocol.
tmp The temporary directory. For each connection, a connection temporary directory with a name like overthere-20111128T132600-7234435.tmp is created within this temporary directory, e.g. /tmp/overthere-20111128T132600-7234435.tmp, to store temporary files for the duration of the connection.
The default value is tmp for UNIX and z/OS hosts and C:\windows\temp for Windows hosts, except for the local protocol where the default is the value of the java.io.tmpdir system property.
tmpFileCreationRetries The number of times Overthere attempts to create a temporary file with a unique name. The default value is 100.
tmpDeleteOnDisconnect If set to false, the connection temporary directory is not removed when the connection. The default value is true.
connectionTimeoutMillis The number of milliseconds Overthere waits for a connection to a remote host to be established. The default value is 120000, i.e. 2 minutes.
socketTimeoutMillis The number of milliseconds Overthere will waits when no data is received on an open connection before raising exception. The default value is 0, i.e. infinite timeout.
jumpstation If set to a non-null value, this property contains the connection options used to connect to an SSH jumpstation (See Tunnelling). Recursive configuration is possible, i.e. this property is also available for the connection options of a jumpstation.
fileCopyCommandForUnix The command to use when copying a file on a Unix host. The string {0} is replaced with the path of the source file, the string {1} is replaced with the path of the destination file. The default value is cp -p {0} {1}.
directoryCopyCommandForUnix The command to use when copying a directory on a Unix host. The string {0} is replaced with the path of the source directory, the string {1} is replaced with the path of the destination directory. The default value is cd {1} ; tar -cf - -C {0} . | tar xpf -. If the tar command is not available but the find command recognizes the -depth parameter with a value, the alternative command find {0} -depth 1 -exec cp -pr {} {1} ; may be configured.
fileCopyCommandForWindows The command to use when copying a file on a Windows host. The string {0} is replaced with the path of the source file, the string {1} is replaced with the path of the destination file. The default value is copy {0} {1} /y.
directoryCopyCommandForWindows The command to use when copying a directory on a Windows host. The string {0} is replaced with the path of the source directory, the string {1} is replaced with the path of the destination directory. The default value is xcopy {0} {1} /i /y /s /e /h /q.
fileCopyCommandForZos The command to use when copying a file on a z/OS host. The string {0} is replaced with the path of the source file, the string {1} is replaced with the path of the destination file. The default value is cp -p {0} {1}.
directoryCopyCommandForZos The command to use when copying a directory on a z/OS host. The string {0} is replaced with the path of the source directory, the string {1} is replaced with the path of the destination directory. The default value is tar cC {0} . | tar xmC {1} .. If the tar command is not available but the find command recognizes the -depth parameter with a value, the alternative command find {0} -depth 1 -exec cp -pr {} {1} ; may be configured.
remoteCopyBufferSize The buffer size to use when copying files from one connection to the other. The buffer size is taken from the _source_ file's connection. The default value is 64 KB (64*1024 bytes). Larger values potentially break copy operations.
remoteCharacterEncoding The character encoding used to transcode files from one connection to the other. The default value is Operating System dependent and is set to 'UTF-8' for Windows and Unix, and to 'Cp1047' (EBCDIC) for Z/OS.

Apart from these common connection options, some protocols define additional protocol-specific connection options. These are documented below, with the corresponding protocol.

LOCAL

The local protocol implementation uses the local file manipulation and local process execution capabilities built-in to Java. The os connection option is hardcoded to the operating system of the local host and the tmp connection option defaults to the system temporary directory as specified by the java.io.tmpdir system property. There are no protocol-specific connection options.

SSH

The SSH protocol implementation of Overthere uses the SSH protocol to connect to remote hosts to manipulate files and execute commands. Most Unix systems already have an SSH server installed and configured and a number of different SSH implementations are available for Windows although not all of them are supported by Overthere.

To connect to a remote host using the SSH protocol, you will need to install an SSH server on that remote host. For Unix and Windows platforms, we recommend OpenSSH. It is included in all Linux distributions and most other Unix flavours. For Windows platforms three SSH servers are supported:

  • OpenSSH on Cygwin. We recommend COPSSH as a convenient packaging of OpenSSH and Cygwin. It is a free source download but since 22/11/2011 the binary installers are a paid solution.
  • WinSSHD is a commercial SSH server that has a lot of configuration options.
  • Install microsoft OpenSSHD on your Windows host and make sure the OpenSSH Server service is enabled.

N.B.: The SFTP, SCP, SU, SUDO and INTERACTIVE_SUDO connection types are only available for Unix hosts. To use SSH with z/OS hosts, use the SFTP connection type. To use SSH with Windows hosts, choose either the SFTP_CYGWIN or the SFTP_WINSSHD or the SFTP_OPENSSHD connection type.

Compatibility

Overthere uses the sshj library for SSH and supports all algorithms and formats supported by that library:

  • Ciphers: aes{128,192,256}-{cbc,ctr}, blowfish-{cbc,ctr}, 3des-{cbc,ctr}, twofish{128,192,256}-{cbc,ctr}, twofish-cbc, serpent{128,192,256}-{cbc,ctr}, idea-{cbc,ctr}, cast128-{cbc,ctr}, arcfour, arcfour{128,256}
  • Key Exchange methods: diffie-hellman-group{1,14}-sha1, diffie-hellman-group-exchange-sha{1,256}, ecdh-sha2-nistp{256,384,521}, [email protected]
  • Signature formats: ssh-rsa, ssh-dss, ecdsa-sha2-nistp256, ssh-ed25519
  • MAC algorithms: hmac-md5, hmac-md5-96, hmac-sha1, hmac-sha1-96, hmac-sha2-256, hmac-sha2-512
  • Compression algorithms: zlib and [email protected] (delayed zlib)
  • Private Key file formats: pkcs8 encoded (the format used by OpenSSH)

SSH host setup

SFTP

To use the SFTP connection type, make sure SFTP is enabled in the SSH server. This is enabled by default in most SSH servers.

SFTP_CYGWIN

To use the SFTP_CYGWIN connection type, install COPSSH on your Windows host. In the COPSSH control panel, add the users as which you want to connect and select Linux shell and Sftp in the shell dropdown box. Check Password authentication and/or Public key authentication depending on the authentication method you want to use.
N.B.: Overthere will take care of the translation from Windows style paths, e.g. C:\Program Files\IBM\WebSphere\AppServer, to Cygwin-style paths, e.g. /cygdrive/C/Program Files/IBM/WebSphere/AppServer, so that your code can use Windows style paths.

SFTP_WINSSHD

To use the SFTP_WINSSHD connection type, install WinSSHD on your Windows host. In the Easy WinSSHD Settings control panel, add the users as which you want to connect, check the Login allowed checkbox and select Allow full access in the Virtual filesystem layout dropdown box. Alternatively you can check the Allow login to any Windows account to allow access to all Windows accounts.
N.B.: Overthere will take care of the translation from Windows style paths, e.g. C:\Program Files\IBM\WebSphere\AppServer, to WinSSHD-style paths, e.g. /C/Program Files/IBM/WebSphere/AppServer, so that your code can use Windows style paths.

SFTP_OpenSSHD

To use the SFTP_OpenSSHD connection type, install OpenSSHD on your Windows host and make sure the OpenSSH Server service is enabled.
N.B.: Overthere will take care of the translation from Windows style paths, e.g. C:\Program Files\IBM\WebSphere\AppServer, to OpenSSHD-style paths, e.g. /C:/Program Files/IBM/WebSphere/AppServer, so that your code can use Windows style paths.

SUDO and INTERACTIVE_SUDO

To use the SUDO connection type, the /etc/sudoers configuration will have to be set up in such a way that the user configured with the connection option username can execute the commands below as the user configured with the connection option sudoUsername. The arguments passed to these commands depend on the exact usage of the Overthere connection. Check the INFO messages on the com.xebialabs.overthere.ssh.SshConnection category to see what commands get executed.

  • chmod
  • cp
  • ls
  • mkdir
  • mv
  • rm
  • rmdir
  • tar
  • Any other command that you want to execute.

The commands mentioned above must be configured with the NOPASSWD setting in the /etc/sudoers file. Otherwise you will have to use the INTERACTIVE_SUDO connection type. When the INTERACTIVE_SUDO connection type is used, every line of the output will be matched against the regular expression configured with the sudoPasswordPromptRegex connection option. If a match is found, the value of the password connection option is sent.

Troubleshooting SSH connections

This section lists a number of common configuration errors that can occur when using Overthere with SSH. If you run into other connectivity issues when using Overthere, pease let us know by creating a ticket or by sending us a pull request.

Cannot start a process on an SSH server because the server disconnects immediately.

If the terminal type requested using the allocatePty connection option or the allocateDefaultPty connection option is not recognized by the SSH server, the connection will be dropped. Specifically, the dummy terminal type configured by [allocateDefaultPty] connection option, will cause OpenSSH on AIX and WinSSHD to drop the connection. Try a safe terminal type such as vt220 instead.

To verify the behaviour of your SSH server with respect to pty allocation, you can manually execute the ssh command with the -T (disable pty allocation) or -t (force pty allocation) flags.

Command executed using SUDO or INTERACTIVE_SUDO fails with the message sudo: sorry, you must have a tty to run sudo

The sudo command requires a tty to run. Set the allocatePty connection option or the allocateDefaultPty connection option to ask the SSH server allocate a pty.

Command executed using SUDO or INTERACTIVE_SUDO appears to hang.

This may be caused by the sudo command waiting for the user to enter his password to confirm his identity. There are two ways to solve this:

  1. Use the NOPASSWD tag in your /etc/sudoers file.
  2. Use the INTERACTIVE_SUDO connection type instead of the SUDO connection type.
  3. If you are already using the INTERACTIVE_SUDO connection type and you still get this error, please verify that you have correctly configured the sudoPasswordPromptRegex option. If you have trouble determining the proper value for the sudoPasswordPromptRegex connection option, set the log level for the com.xebialabs.overthere.ssh.SshElevatedPasswordHandlingStream category to TRACE and examine the output.

SSH connection options

The SSH protocol implementation of Overthere defines a number of additional connection options, in addition to the common connection options.

connectionType Specifies how the SSH protocol is used. One of the following values must be set:
  • SFTP - uses SFTP to transfer files, to a Unix host or a z/OS host, and SSH to execute commands. Requires the SFTP subsystem of the SSH server on the target host to be enabled.
  • SCP - uses SCP to transfer files, to a Unix host, and SSH to execute commands. Can be faster than SFTP, especially over high latency networks.
  • SU - uses SCP to transfer files, to a Unix host. Uses the su command to execute commands. Select this connection type if the username you are connecting with does not have the right permissions to manipulate the files that need to be manipulated and/or to execute the commands that need to be executed.
    If this connection type is selected, the suUsername and suPassword connection option are required and specify the username/password combination that do have the necessary permissions.
  • SUDO - uses SCP to transfer files, to a Unix host. Uses the sudo command, configured with NOPASSWD for all commands, to execute commands. Select this connection type if the username you are connecting with does not have the right permissions to manipulate the files that need to be manipulated and/or to execute the commands that need to be executed.
    If this connection type is selected, the sudoUsername connection option is required and specifies the user that does have the necessary permissions. See below for a more detailed description.
  • INTERACTIVE_SUDO - uses SCP to transfer files, to a Unix host. Uses the sudo command, not been configured with NOPASSWD for all commands, to execute commands. This is similar to the SUDO connection type but also detects the password prompt that is shown by the sudo command when the login user (username) tries to execute a commands as the privileged user (sudoUsername) when that command has not been configured in /etc/sudoers with NOPASSWD.
    N.B.: Because the password of the login user is needed to answer this prompt, this connection type is incompatible with the privateKeyFile option that can be used to authenticate with a private key file.
  • SFTP_CYGWIN - uses SFTP to transfer files, to a Windows host running OpenSSH on Cygwin.
  • SFTP_WINSSHD - uses SFTP to transfer files, to a Windows host running WinSSHD.
  • SFTP_OPENSSHD - uses SFTP to transfer files, to a Windows host running OPENSSHD.
The connection property port specifies the port on which the SSH server listens. The default value for is 22 for all connection types.
suUsername The username of the user that can manipulate the files that need to be manipulated and that can execute the commands that need to be executed.
N.B.: This connection option is only applicable for the SU connection type.
suPassword The password of the user that can manipulate the files that need to be manipulated and that can execute the commands that need to be executed.
N.B.: This connection option is only applicable for the SU connection type.
sudoUsername The username of the user that can manipulate the files that need to be manipulated and that can execute the commands that need to be executed.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
privateKey The RSA private key as String to use when connecting to the remote host. When this connection option is specified, the password and privateKeyFile connection options are ignored.
privateKeyFile The RSA private key file to use when connecting to the remote host. When this connection option is specified, the password connection option is ignored.
passphrase The passphrase to unlock the RSA private key file specified with the privateKeyFile connection option. If this connection option is not specified, the RSA private key file must have an empty passphrase.
allocateDefaultPty If set to true, the SSH server is requested to allocate a default pty for the process, as if the allocatePty option were set to the value dummy:80:24:0:0. The default value is false.
N.B.: This connection option has been deprecated in favour of the allocatePty connection option because it allows the user to specify _what_ pty is allocated.
allocatePty If set, the SSH server is requested to allocate a pty (pseudo terminal) for the process with the setting specified by this option. The format is TERM:COLS:ROWS:WIDTH:HEIGHT, e.g. xterm:80:24:0:0. If set, this option overrides the allocateDefaultPty option.
If the INTERACTIVE_SUDO connection type is used, the default value is vt220:80:24:0:0. Otherwise the default is to not allocate a pty.
All su and sudo implementations require a pty to be allocated for when displaying a password prompt, some sudo implementations even require it when not displaying a password prompt.Some SSH server implementations (notably OpenSSH on AIX 5.3) close the connection when an unknown one is allocated.
heartbeatInterval Specify an interval to send keep-alives packets. Default is 0 (no keep-alive).
interactiveKeyboardAuthRegex The regular expression to look for in keyboard-interactive prompts before sending the password. The default value is .*Password:[ ]?. When the SSH server is configured to not allow password authentication but is configured to allow keyboard-interactive authentication using passwords, Overthere will compare the interactive-keyboard prompt against this regular expression and send the value of the password option when they match. The default value is .*Password:[ ]?
openShellBeforeExecute If set to true, Overthere will open and close a shell immediately before executing a command on an ssh host. This is useful when the connecting user does not yet have a homedir, but this is created for him on the fly on the host. A setup commonly seen when user management is done through LDAP.
suCommandPrefix The command to prefix to the command to be executed to execute it as suUsername. The string {0} is replaced with the value of suUsername. The default value is su - {0} -c.
N.B.: This connection option is only applicable for the SU connection type.
suOverrideUmask If set to true, Overthere will explicitly change the permissions with chmod -R go+rX after uploading a file or directory with scp. The default value is true.
N.B.: This connection option is only applicable for the SU connection type.
suPasswordPromptRegex The regular expression to be used when looking for su password prompts. When the connection type is set to INTERACTIVE_SUDO, Overthere will look for strings that match this regular expression in the first line of the output of a command, and send the password if a match occurs. The default value is .*[Pp]assword.*:
N.B.: This connection option is only applicable for the SU connection type.
suPreserveAttributesOnCopyFromTempFile If set to true, files are copied from the connection temporary directory using the -p flag to the cp command. The default value is false.
N.B.: This connection option is only applicable for the SU connection type.
suPreserveAttributesOnCopyToTempFile If set to true, files are copied to the connection temporary directory using the -p flag to the cp command. The default value is false.
N.B.: This connection option is only applicable for the SU connection type.
suQuoteCommand If set to true, the original command is added as one argument to the prefix configured with the suCommandPrefix connection option. This has the result of quoting the original command, which is needed for commands like su. Compare su -u privilegeduser start server1 to su privilegeduser -c 'start server1'. The default value is true.
N.B.: This connection option is only applicable for the SU connection type.
sudoCommandPrefix The command to prefix to the command to be executed to execute it as sudoUsername. The string {0} is replaced with the value of sudoUsername. The default value is sudo -u {0}.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoOverrideUmask If set to true, Overthere will explicitly change the permissions with chmod -R go+rX after uploading a file or directory with scp. The default value is true.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoPasswordPromptRegex The regular expression to be used when looking for sudo password prompts. When the connection type is set to INTERACTIVE_SUDO, Overthere will look for strings that match this regular expression in the first line of the output of a command, and send the password if a match occurs. The default value is .*[Pp]assword.*:
N.B.: This connection option is only applicable for the INTERACTIVE_SUDO connection type.
sudoPreserveAttributesOnCopyFromTempFile If set to true, files are copied from the connection temporary directory using the -p flag to the cp command. The default value is false.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoPreserveAttributesOnCopyToTempFile If set to true, files are copied to the connection temporary directory using the -p flag to the cp command. The default value is false.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoQuoteCommand If set to true, the original command is added as one argument to the prefix configured with the sudoCommandPrefix connection option. This has the result of quoting the original command, which is needed for commands like su. Compare sudo -u privilegeduser start server1 to su privilegeduser -c 'start server1'. The default value is false.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
deleteDirectoryCommand The command to be used when deleting a directory. The string {0} is replaced with the value of the path of the directory to be deleted. The default value is rmdir {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
deleteFileCommand The command to be used when deleting a file. The string {0} is replaced with the value of the path of the file to be deleted. The default value is rm -f {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
deleteRecursivelyCommand The command to be used when deleting a directory recursively. The string {0} is replaced with the value of the path of the directory to be deleted. The default value is rm -rf {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
getFileInfoCommand The command to be used when getting the metadata of a file/directory. The string {0} is replaced with the value of the path of the file/directory. The default value is ls -ld {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
listFilesCommand The command to be used when listing the contents of a directory. The string {0} is replaced with the value of the path of the directory to be listed. The default value is ls -a1 {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
mkdirCommand The command to be used when creating a directory. The string {0} is replaced with the value of the path of the directory to be created. The default value is mkdir {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
mkdirsCommand The command to be used when creating a directory tree. The string {0} is replaced with the value of the path of the directory tree to be created. The default value is mkdir -p {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
renameToCommand The command to be used when renaming a file/directory. The string {0} is replaced with the value of the path of the file/directory to be renamed. The string {1} is replaced with the value of the new name. The default value is mv {0} {1}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
setExecutableCommand The command to be used when making a file executable. The string {0} is replaced with the value of the path of the file/directory affected. The default value is chmod a+x {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
setNotExecutableCommand The command to be used when making a file non-executable. The string {0} is replaced with the value of the path of the file/directory affected. The default value is chmod a-x {0}.
N.B.: This connection option is only applicable for the SCP, SUDO and INTERACTIVE_SUDO connection types.
sudoTempMkdirCommand The command to be used when creating a temporary directory as a sudo user. The directory needs to be read/writeable for both the connecting and the sudo user. The string {0} is replaced with the value of the path of the directory to be created. The default value is mkdir -m 1777 {0}.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoTempMkdirsCommand The command to be used when creating a temporary directory tree as a sudo user. The directory tree needs to be read/writeable for both the connecting and the sudo user. The string {0} is replaced with the value of the path of the directory to be created. The default value is mkdir -p -m 1777 {0}.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoCopyFromTempFileCommand The command to be used when copying files/directories from the connection temporary directory as the sudo user. The string {0} is replaced with the value of the path of the file/directory being copied. The string {1} is replaced with the value of the target path. The default value is cp -pr {0} {1} if sudoPreserveAttributesOnCopyFromTempFile is set to true, otherwise the default value is cp -r {0} {1}.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoOverrideUmaskCommand The command to be used when setting the umask before copying a file/directory from, or after copying it to the connection temporary directory. This command ensures that the sudo user has read (and/or execute) rights for the copied file/directory. The string {0} is replaced with the value of the file/directory being copied. The default value is chmod -R go+rX {0}.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
sudoCopyToTempFileCommand The command to be used when copying files/directories to the connection temporary directory as the sudo user. The string {0} is replaced with the value of the path of the file/directory being copied. The string {1} is replaced with the value of the target path. The default value is cp -pr {0} {1} if sudoPreserveAttributesOnCopyToTempFile is set to true, otherwise the default value is cp -r {0} {1}.
N.B.: This connection option is only applicable for the SUDO and INTERACTIVE_SUDO connection types.
localAddress The address to use on the local machine as the source address of the connection. This property is optional and mainly useful on systems with more than one address. The default behaviour is to let the OS decide.
localPort The port to use on the local machine as the source port of the connection. This property is optional and to be used in combination with the localAddress property. The default behaviour is to let the OS pick a free port.
transportTimeoutMillis The number of milliseconds Overthere will wait for exchanging SSH keys and to authenticate user. The default value is 30000, i.e. 30s timeout.

SMB 2.x/CIFS, WinRM and Telnet

SMB 2.x

The SMB 2.x protocol implementation of Overthere uses the prevalent SMB protocol for file manipulation and, depending on the settings, using either WinRM or Telnet for process execution.

CIFS

The CIFS protocol implementation of Overthere uses public variant of the original Server Message Block (SMB) protocol developed by Microsoft known as CIFS protocol, for file manipulation and, depending on the settings, uses either WinRM or Telnet for process execution. CIFS is widely regarded as an obsolete protocol and users are encouraged to prefer SMB protocol over CIFS. Support for CIFS is deprecated and will be removed from subsequent releases.

SMB 2.x and CIFS

These protocols are only supported for Windows hosts, you will most likely not need to install new software although you might need to enable and configure some services:

  • The built-in file sharing capabilities of Windows are based on CIFS/SMB and are therefore available and enabled by default.
  • WinRM is available on Windows Server 2008 and up. Overthere supports basic authentication for local accounts and Kerberos authentication for domain accounts. Overthere has a built-in WinRM library that can be used from all operating systems by setting the connectionType connection option to WINRM_INTERNAL. When connecting from a host that runs Windows, or when using a "winrs proxy host" that runs Windows, the native WinRM capabilities of Windows, i.e. the winrs command, can be used by setting the connectionType connection option to WINRM_NATIVE.
  • A Telnet Server is available on all Windows Server versions although it might not be enabled.

To connect to a remote host using the SMB or CIFS protocol, ensure the host is reachable on port 445.

If you will be connecting as an administrative user, ensure the administrative shares are configured. Otherwise, ensure that the user you will be using to connect has access to shares that correspond to the directory you want to access and that the pathShareMappings connection option is configured accordingly.

Password limitations

Due to a limitation of the winrs command, passwords containing a single quote (') or a double quote (") cannot be used when using the WINRM_NATIVE connection type.

Domain accounts

Windows domain accounts are supported by the WINRM_INTERNAL, WINRM_NATIVE and TELNET connection types, but the syntax of the username is different:

  • For the WINRM_INTERNAL connection type, domain accounts must be specified using the new-style domain syntax, e.g. [email protected].
  • For the TELNET connection type, domain accounts must be specified using the old-style domain syntax, e.g DOMAIN\USER.
  • For the WINRM_NATIVE connection type, domain accounts may be specified using either the new-style ([email protected]) or old-style (DOMAIN\USER) domain syntax.
  • For all three connection types, local accounts must be specified without an at-sign (@) or a backslash (\).

N.B.: When using domain accounts with the WINRM_INTERNAL connection type, the Kerberos subsystem of the Java Virtual Machine must be configured correctly. Please read the section on how to set up Kerberos for the source host and the remote hosts.

Administrative shares

By default Overthere will access the administrative shares on the remote host. These shares are only accessible for users that are part of the Administrators on the remote host. If you want to access the remote host using a regular account, use the pathShareMapping connection option to configure the shares to use for the paths Overthere will be connecting to. Of course, the user configured with the username connection option should have access to those shares and the underlying directories and files.

N.B.: Overthere will take care of the translation from Windows paths, e.g. C:\Program Files\IBM\WebSphere\AppServer, to SMB URLs that use the administrative shares, e.g. smb://username:password@hostname/C$/Program%20Files/IBM/WebSphere/AppServer (which corresponds to the UNC path \\hostname\C$\Program Files\IBM\WebSphere\AppServer), so that your code can use Windows style paths.

Host setup

TELNET

To use the TELNET connection type, you'll need to enable and configure the Telnet Server according to these instructions:

  1. (Optional) If the Telnet Server is not already installed on the remote host, add it using the Add Features Wizard in the Server Manager console.

  2. (Optional) If the remote host is running Windows Server 2003 SP1 or an x64-based version of Windows Server 2003, install the Telnet server according to these instructions from the Microsoft Support site.

  3. Enable the Telnet Server Service on the remote host according to these instructions on the Microsoft Technet site.

  4. After you have started the Telnet Server, open a command prompt as the Administrator user on the remote host and enter the command tlntadmn config mode=stream to enable stream mode.

When the Telnet server is enabled any user that is in the Administrators group or that is in the TelnetClients group and that has the Allow logon locally privilege can log in using Telnet. See the Microsoft Technet to learn how to grant a user or group the right to logon locally on Windows Server 2008 R2.

WINRM (WINRM_INTERNAL and WINRM_NATIVE)

For a PowerShell script to do what is described below in one go, check Richard Downer's blog

To use the WINRM_INTERNAL or the WINRM_NATIVE connection type, you'll need to setup WinRM on the remote host by following these instructions:

  1. If the remote host is running Windows Server 2003 SP1 or SP2, or Windows XP SP2, install the WS-Management v.1.1 package.

  2. If the remote host is running Windows Server 2003 R2, go to the Add/Remove System Components feature in the Control Panel and add WinRM under the section Management and Monitoring Tools. Afterwards install the WS-Management v.1.1 package to upgrade the WinRM installation.

  3. If the remote host is running Windows Vista or Windows 7, the Windows Remote Management (WS-Management) service is not started by default. Start the service and change its Startup type to Automatic (Delayed Start) before proceeding with the next steps.

  4. On the remote host, open a Command Prompt (not a PowerShell prompt!) using the Run as Administrator option and paste in the following lines when using the WINRM_INTERNAL connection type:

     winrm quickconfig
 y
 winrm set winrm/config/service/Auth @{Basic="true"}
 winrm set winrm/config/service @{AllowUnencrypted="true"}
 winrm set winrm/config/winrs @{MaxMemoryPerShellMB="1024"}

    Or the following lines when using the WINRM_NATIVE connection type:

     winrm quickconfig
 y
 winrm set winrm/config/service/Auth @{Basic="true"}
 winrm set winrm/config/winrs @{MaxMemoryPerShellMB="1024"}

    Or keep reading for more detailed instructions. :-)

  5. Run the quick config of WinRM to start the Windows Remote Management service, configure an HTTP listener and create exceptions in the Windows Firewall for the Windows Remote Mangement service:

     winrm quickconfig

    N.B.: The Windows Firewall needs to be running to run this command. See Microsoft Knowledge Base article #2004640.

  6. (Optional) By default basic authentication is disabled in WinRM. Enable it if you are going to use local accounts to access the remote host:

     winrm set winrm/config/service/Auth @{Basic="true"}

  7. (Optional) By default Kerberos authentication is enabled in WinRM. Disable it if you are not going to use domain accounts to access the remote host:

     winrm set winrm/config/service/Auth @{Kerberos="false"}

    N.B.: Do not disable Negotiate authentication as the winrm command itself uses that to configure the WinRM subsystem!

  8. (Only required for WINRM_INTERNAL or when the connection option winrsUnencrypted is set to true) Configure WinRM to allow unencrypted SOAP messages:

     winrm set winrm/config/service @{AllowUnencrypted="true"}

  9. Configure WinRM to provide enough memory to the commands that you are going to run, e.g. 1024 MB:

     winrm set winrm/config/winrs @{MaxMemoryPerShellMB="1024"}

    N.B.: This is not supported by WinRM 3.0, included with the Windows Management Framework 3.0. This update has been temporarily removed from Windows Update because of numerous incompatiblity issues with other Microsoft products. However, if you have already installed WMF 3.0 and cannot downgrade, Microsoft Knowledge Base article #2842230 describes a hotfix that can be installed to re-enable the MaxMemoryPerShellMB setting.

  10. To use the WINRM_INTERNAL or WINRM_NATIVE connection type with HTTPS, i.e. winrmEnableHttps set to true, follow the steps below:

    1. (Optional) Create a self signed certificate for the remote host by installing selfssl.exe from the IIS 6 resource kit and running the command below or by following the instructions in this blog by Hans Olav:

       C:\Program Files\IIS Resources\SelfSSL>selfssl.exe /T /N:cn=HOSTNAME /V:3650
 Microsoft (R) SelfSSL Version 1.0
 Copyright (C) 2003 Microsoft Corporation. All rights reserved.

 Do you want to replace the SSL settings for site 1 (Y/N)?Y
 The self signed certificate was successfully assigned to site 1.

    2. Open a PowerShell window and enter the command below to find the thumbprint for the certificate for the remote host:

       PS C:\Windows\system32> Get-childItem cert:\LocalMachine\Root\ | Select-String -pattern HOSTNAME

 [Subject]
   CN=HOSTNAME

 [Issuer]
   CN=HOSTNAME

 [Serial Number]
   527E7AF9142D96AD49A10469A264E766

 [Not Before]
   5/23/2011 10:23:33 AM

 [Not After]
   5/20/2021 10:23:33 AM

 [Thumbprint]
   5C36B638BC31F505EF7F693D9A60C01551DD486F

    3. Create an HTTPS WinRM listener for the remote host with the thumbprint you've just found:

       winrm create winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname="HOSTNAME"; CertificateThumbprint="THUMBPRINT"}

For more information on WinRM, please refer to the online documentation at Microsoft's DevCenter. As a quick reference, have a look at the list of useful commands below:

  • Do a quickconfig for WinRM with HTTPS: winrm quickconfig -transport:https
  • View the complete WinRM configuration: winrm get winrm/config
  • View the listeners that have been configured: winrm enumerate winrm/config/listener
  • Create an HTTP listener: winrm create winrm/config/listener?Address=*+Transport=HTTP (also done by winrm quickconfig)
  • Allow all hosts to connect to the WinRM listener: winrm set winrm/config/client @{TrustedHosts="*"}
  • Allow a fixed set of hosts to connect to the WinRM listener: winrm set winrm/config/client @{TrustedHosts="host1,host2..."}

Kerberos - source host

N.B.: You will only need to configure Kerberos if you are going to use Windows domain accounts to access the remote host with the WINRM_INTERNAL connection type.

In addition to the setup described in the WINRM section, using Kerberos authentication requires that you follow the Kerberos Requirements for Java on the host from which the Overthere connections are initiated, i.e. the source host.

Create a file called krb5.conf (Unix) or krb5.ini (Windows) with at least the following content:

[realms]
EXAMPLE.COM = {
    kdc = KDC.EXAMPLE.COM
}

Replace the values with the name of your domain/realm and the hostname of your domain controller (multiple entries can be added to allow the Overthere source host to connect to multiple domains) and place the file in the default location for your operating system:

  • Linux: /etc/krb5.conf
  • Solaris: /etc/krb5/krb5.conf
  • Windows: C:\Windows\krb5.ini

Alternatively, place the file somewhere else and add the following Java system property to the command line: -Djava.security.krb5.conf=/path/to/krb5.conf. Replace the path with the location of the file you just created.

See the Kerberos V5 System Administrator's Guide at MIT for more information on the krb5.conf format.

Kerberos - remote host

N.B.: You will only need to configure Kerberos if you are going to use Windows domain accounts to access the remote host with the WINRM_INTERNAL connection type.

By default, Overthere 2.1.0 and up will request access to a Kerberos service principal name of the form WSMAN/HOST, for which an SPN should be configured automatically when you configure WinRM for a remote host.

If that was not configured correctly, if you have overridden the default SPN for which a ticket is requested through the winrmKerberosAddPortToSpn or the winrmKerberosUseHttpSpn connection properties, or if you are running an older version of Overthere, you will have configure the service principal names manually. The parameter winrmUseCanonicalHostname can be used to use the remote host FQDN provided by the DNS.

This can be achieved by invoking the setspn command, as an Administrator, on any host in the domain, as follows:

setspn -A PROTOCOL/ADDRESS:PORT WINDOWS-HOST

where:

  • PROTOCOL is either WSMAN (default) or HTTP (if winrmKerberosUseHttpSpn has been set to true).
  • ADDRESS is the address used to connect to the remote host,
  • PORT (optional) is the port used to connect to the remote host (usually 5985 or 5986, only necessary if winrmKerberosAddPortToSpn has been set to true), and
  • WINDOWS-HOST is the short Windows hostname of the remote host.

Some other useful commands:

  • List all service principal names configured for the domain: setspn -Q */*
  • List all service principal names configured for a specific host in the domain: setspn -L _WINDOWS-HOST_

Troubleshooting SMB 2.x/CIFS, WinrRM and Telnet

This section lists a number of common configuration errors that can occur when using Overthere with SMB/CIFS, WinRM and/or Telnet. If you run into other connectivity issues when using Overthere, please let us know by creating a ticket or by sending us a pull request.

For more troubleshooting tips for Kerberos, please refer to the Kerberos troubleshooting guide in the Java SE documentation.

Kerberos authentication fails with the message Unable to load realm info from SCDynamicStore

The Kerberos subsystem of Java cannot start up. Did you configure it as described in the section on Kerberos setup for the source host?

Kerberos authentication fails with the message Cannot get kdc for realm …

The Kerberos subsystem of Java cannot find the information for the realm in the krb5.conf file. The realm name specified in the Kerberos configuration on the source host is case sensitive and must be entered in upper case in the krb5.conf file.

Alternatively, you can use the dns_lookup_kdc and dns_lookup_realm options in the libdefaults section to automatically find the right realm and KDC from the DNS server if it has been configured to include the necessary SRV and TXT records:

[libdefaults]
    dns_lookup_kdc = true
    dns_lookup_realm = true

Kerberos authentication fails with the message Server not found in Kerberos database (7)

The service principal name for the remote host has not been added to Active Directory. Did you add the SPN as described in the section on Kerberos setup for remote hosts?

Kerberos authentication fails with the message Pre-authentication information was invalid (24) or Identifier doesn't match expected value (906)

The username or the password supplied was invalid. Did you supply the correct credentials?

Kerberos authentication fails with the message Integrity check on decrypted field failed (31)

Is the target machine part of a Windows 2000 domain? In that case, you'll have to add rc4-hmac to the supported encryption types:

[libdefaults]
    default_tgs_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4 rc4-hmac
    default_tkt_enctypes = aes256-cts-hmac-sha1-96 des3-cbc-sha1 arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des-cbc-md4 rc4-hmac

Kerberos authentication fails with the message Message stream modified (41)

The realm name specified in the Kerberos configuration on the source host does not match the case of the Windows domain name. The realm name is case sensitive and must be entered in upper case in the krb5.conf file.

I am not using Kerberos authentication and I still see messages saying Unable to load realm info from SCDynamicStore

The Kerberos subsystem of Java cannot start up and the remote WinRM server is sending a Kerberos authentication challenge. If you are using local accounts, the authentication will proceed succesfully despite this message. To remove these messages either configure Kerberos as described in the section on Kerberos setup for the source host or disallow Kerberos on the WinRM server as described in step 4 of the section on WinRM setup.

Telnet connection fails with the message VT100/ANSI escape sequence found in output stream. Please configure the Windows Telnet server to use stream mode (tlntadmn config mode=stream).

The Telnet service has been configured to be in "Console" mode. Did you configure it as described in the section on Telnet setup?

The winrm configuration command fails with the message There are no more endpoints available from the endpoint mapper

The Windows Firewall has not been started. See Microsoft Knowledge Base article #2004640 for more information.

The winrm configuration command fails with the message The WinRM client cannot process the request

This can occur if you have disabled the Negotiate authentication method in the WinRM configuration. To fix this situation, edit the configuration in the Windows registry under the key HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\ and restart the Windows Remote Management service.

Courtesy of this blog by Chris Knight

WinRM command fails with the message java.net.ConnectException: Connection refused

The Windows Remote Management service is not running or is not running on the port that has been configured. Start the service or configure Overthere to use a different port number.

WinRM command fails with a 401 response code

Multiple causes can lead to this error message:

  1. The Kerberos ticket is not accepted by the remote host:

    1. Did you set up the correct service principal names (SPNs) as described in the section on Kerberos setup for remote hosts? The hostname is case insenstive, but it has to be the same as the one used in the address connection options, i.e. a simple hostname or a fully qualified domain name. Domain policies may prevent the Windows Management Service from creating the required SPNs. See this blog by LazyJeff for more information.

    2. Has the reverse DNS of the remote host been set up correctly? See Principal names and DNS for more information. Please note that the rdns option is not available in Java's Kerberos implementation.

  2. The WinRM service is not set up to accept unencrypted traffic. Did you execute step #8 of the host setup for WinRM?

  3. The user is not allowed to log in. Did you uncheck the "User must change password at next logon" checkbox when you created the user in Windows?

  4. The user is not allowed to perform a WinRM command. Did you grant the user (local) administrative privileges?

WinRM command fails with a 500 response code

Multiple causes can lead to this error message:

  1. If the command was executing for a long time, this might have been caused by a timeout. You can increase the WinRM timeout specified by the winrmTimeout connection option to increase the request timeout. Don't forget to increase the MaxTimeoutms setting on the remote host as well. For example, to set the maximum timeout on the server to five minutes, enter the following command:

    winrm set winrm/config @{MaxTimeoutms="300000"}

  2. If a lot of commands are being executed concurrently, increase the MaxConcurrentOperationsPerUser setting on the server. For example, to set the maximum number of concurrent operations per user to 100, enter the following command:

    winrm set winrm/config/service @{MaxConcurrentOperationsPerUser="100"}

Other configuration options that may be of use are Service/MaxConcurrentOperations and MaxProviderRequests (WinRM 1.0 only).

WinRM command fails with an unknown error code

If you see an unknown WinRM error code in the logging, you can use the winrm helpmsg command to get more information, e.g.

winrm helpmsg 0x80338104
The WS-Management service cannot process the request. The WMI service returned an 'access denied' error.

Courtesy of this PowerShell Magazine blog by Shay Levy

Troubleshooting CIFS

CIFS connections are very slow to set up.

The JCIFS library, which Overthere uses to connect to CIFS shares, will try and query the Windows domain controller to resolve the hostname in SMB URLs. JCIFS will send packets over port 139 (one of the [NetBIOS over TCP/IP] ports) to query the DFS. If that port is blocked by a firewall, JCIFS will only fall back to using regular hostname resolution after a timeout has occurred.

Set the following Java system property to prevent JCIFS from sending DFS query packets: -Djcifs.smb.client.dfs.disabled=true.

See this article on the JCIFS mailing list for a more detailed explanation.

CIFS connections time out

If the problem cannot be solved by changing the network topology, try increasing the JCIFS timeout values documented in the JCIFS documentation. Another system property not mentioned there but only on the JCIFS homepage is jcifs.smb.client.connTimeout.

To get more debug information from JCIFS, set the system property jcifs.util.loglevel to 3.

Common SMB 2.x and CIFS connection options

The SMB 2.x and CIFS protocol implementation of Overthere defines a number of additional connection options, in addition to the common connection options.

connectionType Specifies what protocol is used to execute commands on the remote hsots. One of the following values must be set:
  • WINRM_INTERNAL - uses WinRM over HTTP(S) to execute remote commands. The port connection option specifies the Telnet port to connect to. The default value is 5985 for HTTP and 5986 for HTTPS. A Java implementation of WinRM used.
  • WINRM_NATIVE - uses WinRM over HTTP(S) to execute remote commands. The port connection option specifies the Telnet port to connect to. The default value is 5985 for HTTP and 5986 for HTTPS. The native Windows implementation of WinRM is used, i.e. the winrs command.
  • TELNET - uses Telnet to execute remote commands. The port connection option specifies the Telnet port to connect to. The default value is 23.
pathShareMappings The path to share mappings to use for CIFS specified as a Map<String, String>, e.g. C:\IBM\WebSphere -> WebSphere. If a path is not explicitly mapped to a share, an administrative share will be used. The default value is to use no path/share mappings, i.e. to use only administrative shares.
winrmEnableHttps If set to true, HTTPS is used to connect to the WinRM server. Otherwise HTTP is used. The default value is false.
N.B.: This connection option is only applicable for the WINRM_INTERNAL and WINRM_NATIVE connection types.
winrmContext The context used by the WinRM server. The default value is /wsman.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type.
winrmEnvelopSize The WinRM envelop size in bytes to use. The default value is 153600.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type.
winrmHttpsCertificateTrustStrategy The certificate trust strategy for WinRM HTTPS connections. One of the following values can be set:
  • STRICT (default) - use Java's trusted certificate chains.
  • SELF_SIGNED - self-signed certificates are allowed (see TrustSelfSignedStrategy)
  • ALLOW_ALL - trust all certificates.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type, when winrmEnableHttps is set to true.
winrmHttpsHostnameVerificationStrategy The hostname verification strategy for WinRM HTTPS connections. One of the following values can be set: See the Apache HttpComponent HttpClient documentation for more information about the hostname verifications strategies.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type, when winrmEnableHttps is set to true.
winrmKerberosAddPortToSpn If set to true, the port number (e.g. 5985) will be added to the service principal name (SPN) for which a Kerberos ticket is requested. The default value is false.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type, when a Windows domain acount is used.
winrmKerberosDebug If set to true, enables debug output for the JAAS-based Kerberos authentication within the OverThere connector. The default value is false.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type, when a Windows domain acount is used.
winrmKerberosTicketCache If set to true, enables the use of the Kerberos ticket cache for use in authentication. When enabled, if a password is not specfified the system ticket cache will be used as a The default value is false.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type, when a Windows domain acount is used.
winrmKerberosUseHttpSpn If set to true, the protocol HTTP will be used in the service principal name (SPN) for which a Kerberos ticket is requested. Otherwise the protocol WSMAN is used. The default value is false.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type, when a Windows domain acount is used.
winrmLocale The WinRM locale to use. The default value is en-US.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type.
winrmTimeout The WinRM timeout to use in XML schema duration format. The default value is PT60.000S.
N.B.: This connection option is only applicable for the WINRM_INTERNAL connection type.
winrsAllowDelegate If set to false, the user's credentials may be passed to the remote host. This option corresponds to the winrs command option -allowdelegate. The default value is false.
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.
winrsCompression If set to true, compression is enabled. This option corresponds to the winrs command option -compression. The default value is false.
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.
winrsNoecho If set to true, echo is disabled. This option corresponds to the winrs command option -noecho. The default value is false.
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.
winrsNoprofile If set to true, loading the user profile before executing the command is disabled. This option corresponds to the winrs command option -noprofile. The default value is false.
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.
winrsUnencrypted If set to true, encryption is disabled. This option corresponds to the winrs command option -unencrypted. The default value is false.
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.
winrsProxyProtocol The protocol to use when connecting to the "winrs proxy host", i.e. the host that is used to run the winrs command. The "winrs proxy host" must run Windows. The default value is local, which means the commands will be executed on the local host, which means the local host must run Windows.
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.
winrsProxyConnectionOptions The connection options to use when connecting to the "winrs proxy host".
N.B.: This connection option is only applicable for the WINRM_NATIVE connection type.

SMB 2.x connection options

smbPort The SMB port to connect to. The default value is 445.
smbRequireSigning Whether to require the server to sign the responses. The default value is false.

CIFS connection options

cifsPort The CIFS port to connect to. The default value is 445.

Jumpstations: SSH tunnels and HTTP proxies

In some networks, certain hosts cannot be reached directly. Instead, a connection should be made through an HTTP proxy or an SSH jumpstation. Overthere supports these network topologies by allowing you to configure "jumpstations" by adding a property called jumpstation to the connection options that point to another set of connection options for the jumpstation.

The end result looks something like this:

protocol: ssh
options: ConnectionOptions[
	connectionType: SCP
	address: dmzhost1.example.com
	os: UNIX
	username: dmzuser
	password: s3cr3t
	jumpstation: ConnectionOptions[
		protocol: proxy
		address: dmzproxy.exmaple.com
		port: 8888
	]
]

or:

protocol: cifs
options: ConnectionOptions[
	connectionType: WINRM_INTERNAL
	address: dmzhost2.example.com
	os: WINDOWS
	username: Administrator
	password: adm1n=me
	jumpstation: ConnectionOptions[
		protocol: ssh-jumpstation
		address: dmzssh.example.com
		username: jumpuser
		password: j0mp!
	]
]

The jumpstation connection options support the same values (for as much as it makes sense) as regular connection options, with the following additions:

protocol The kind of jumpstation to use. The following values can be used:
  • proxy - use an HTTP proxy
  • ssh-jumpstation (or ssh)- set up an SSH tunnel with a local port forwarding
The default vaue is ssh-jumpstation.
portAllocationRangeStart The port number Overthere starts with to find an available local port for setting up an SSH local port forwarder. This option only applies when using the ssh-jumpstation protocol. The default value is 1024.

SSH connection to GCP instances

SSH connection to GCP instances is possible without directly installing key pairs on instances. There are 2 options to manage internally SSH keys via GCP keys, that are selected with gcpKeyManagementType.

gcpKeyManagementType Currently folllowing 2 types are suppport for key management on GCP instances:
  • `OsLogin` - managing ssh keys in with OsLogin API;
  • `Metadata` - managing ssh keys by changing value under a metadata key `ssh-keys`

GCP OsLogin: Using GCP's service account with OsLogin

SSH connection to the GCP hosts can be implemented by managing ssh keys with OsLogin API. It gives the possibility to define temporary time-limited SSH keys project-wide or per an instance by using GCP's service account credentials. OsLogin management of SSH keys is recommended way to connect on instances, but it has some limitations.

Prerequisites:

  1. Get service account key to JSON file (for ServiceAccountCredentials)
  2. Enable OS Login step 3
  3. Target instance with a role (roles/compute.osAdminLogin or roles/compute.osLogin) on project or instance level step 4

GCP Metadata: Using GCP's service account with metadata

Metadata gives the possibility to define temporary time-limited SSH keys project-wide or per instance. Overthere manages keypairs internally registering public key to the GCP metadata on project or instance level. More details is available in following guide: Managing SSH keys in metadata.

Prerequisites:

  1. Get service account key to JSON file (for ServiceAccountCredentials)
  2. Service account needs to have the following IAM roles on the project:
    • iam.serviceAccountUser on the instance or project level
    • compute.instanceAdmin.v1 on the instance or project level
  3. Target instance with a role (roles/compute.osAdminLogin or roles/compute.osLogin) on project or instance level

For using Metadata following options are required:

zoneName Name of deployment area. See Regions and zones for details.
instanceId Instance id of a instance that needs management of the keys. If managing of the keys is done on a project level, this option needs to be omitted.
applicationName Optional application name that is using this overthere client.
username SSH connection username. It is same options as for SSH connection

GCP Generic SSH options

From following options first 3 are required. Other generic options have default value that is used if custom not provided.

connectionType Specifies which protocol is used to execute commands, it can be any SSH related protocol like: `SCP`, `SFTP`
os Same as with usual SSH connection
address On the GCP external instance address. Same as with usual SSH connection.
retryCount Retry count of trying to connect to the instance. Connection openning retry will be done in case of auth failure. Key management on GCP needs sometime more time to provision new keys on instance level and for that case failed auth needs retries. Default value is 10.
retryPeriodMillis Retry period between each retry to open the connection. Default value is 1000 ms.
keySize Key size, default is 1024.
keyExpiryTimeMillis Key expiry time defined in milliseconds, default value is 300,000ms (5 minutes). If generated key expires new one will be internally created and installed via OsLogin.

GCP Credentials

To be able to use GCP API's credentials needs to be supplied via Overthere options. There are different ways to supply credentials and that is defined with the gcpCredentialsType option.

gcpCredentialsType
  • `Default` - The following are searched (in order) to find the Application Default Credentials:
    1. Credentials file pointed to by the `GOOGLE_APPLICATION_CREDENTIALS` environment variable
    2. Credentials provided by the Google Cloud SDK `gcloud auth application-default login` command
    3. Google App Engine built-in credentials
    4. Google Cloud Shell built-in credentials
    5. Google Compute Engine built-in credentials
  • `ServiceAccountJsonFile` - defined by a Service Account key file in JSON format from the Google Developers Console.
  • `ServiceAccountJson` - defined by a Service Account key in JSON format from the Google Developers Console.
  • `ServiceAccountPkcs8` - defined by a Service Account PKCS#8 private key and other required options.
  • `ServiceAccountToken` - defined by an Access Token.

For ServiceAccountJsonFile following option is required:

credentialsFile Path to the service account credentials file. Credentials are used to internally manage SSH keys. The easiest way to get it is via gcloud command: 
gcloud iam service-accounts keys create path_to_credentials_json \
--iam-account $SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com

For ServiceAccountJson following option is required:

credentialsJson Service account credentials JSON. Credentials are used to internally manage SSH keys.

For ServiceAccountPkcs8 following options are required:

projectId Project ID
clientId Client ID of the service account
clientEmail Client email address of the service account
privateKeyPkcs8 RSA private key object for the service account in PKCS#8 format.
privateKeyId Private key identifier for the service account.
scopes Scopes, comma separated list, for the APIs to be called. May be empty, which results in a credential that must have createScoped called before use.
tokenServerUri URI of the end point that provides tokens.
serviceAccountUser The email of the user account to impersonate, if delegating domain-wide authority to the service account.

For ServiceAccountToken following option is required:

projectId Project ID
ApiToken Access Token Which can be used for Authentication.Generated by using the Token URL.

Release History

  • Overthere 5.3.2 (30-Mar-2021)
    • SSH connection to GCP hosts by using project or instance metadata.
    • Fix SSH connection resource leak on connection failure.
  • Overthere 5.3.1 (26-Mar-2021)
    • Add transportTimeoutMillis option for ssh connection.
  • Overthere 5.3.0 (23-Mar-2021)
    • SSH connection to GCP hosts by using OsLogin API.
  • Overthere 5.2.1 (10-Dec-2020)
    • Supported Opensshd service for windows host by SFTP_Opensshd connection type.
  • Overthere 5.0.21 (07-Oct-2020)
    • Upgrade to SMBJ 0.10.0
  • Overthere 5.0.20 (18-Sep-2020)
    • Fix for #247 (Make artifact-copy-strategy work on satellite)
  • Overthere 5.0.10 (29-Jul-2020)
    • Fix for #245 (Execution of cmd.command that contains percentage symbol(%) on windows)
  • Overthere 5.0.9 (16-Jul-2020)
    • Fix for #244 (Escaping % character in passwords breaks winrm_native authentication)
  • Overthere 5.0.8 (30-Jun-2020)
    • Upgrade to dom4j 2.1.3
  • Overthere 5.0.7 (16-Jan-2020)
    • Improved error handling on file delete failure
    • Improved logging on closing of streams
  • Overthere 5.0.6 (5-Aug-2019)
    • Upgrade to dom4j 2.1.1
  • Overthere 5.0.5 (23-Jul-2019)
    • Improved Windows command line sanitization
  • Overthere 5.0.4 (1-Mar-2019)
    • Fixes for #231, #232
    • Upgrade to SSHJ 0.27.0, SMBJ 0.9.1
  • Overthere 5.0.3 (22-Aug-2018)
    • Fixes for #207, #211, #212, #213, #218
    • Upgrade to SSHJ 0.26.0, SMBJ 0.8.0
    • Support programmatic protocol registration
  • Overthere 5.0.2 (30-Aug-2017)
    • Upgraded to SMBJ 0.3.0
    • Added support for transcode OvertherFile to another encoding
    • Added support for transparent tunneling for SMB
  • Overthere 5.0.1 (08-Jun-2017)
    • Mapped diagnostic context of the parent thread to its child.
  • Overthere 5.0.0 (21-Mar-2017)
    • Upgraded to SSHJ 0.21.1
      • SSHJ 0.21.1 (2017-04-25)
      • Fix regression from 40f956b (invalid length parameter on outputstream)
      • SSHJ 0.21.0 (2017-04-14)
      • Added support for [email protected] and [email protected] certificate key files
      • Upgraded Gradle to 3.4.1
      • Added support for custom string encoding
      • Fixed #312: Upgraded BouncyCastle to 1.56
  • Overthere 4.4.6 (21-Mar-2017)
    • Upgraded to SMBJ 0.0.9
  • Overthere 4.4.5 (17-Mar-2017)
    • Upgraded to SMBJ 0.0.8
  • Overthere 4.4.4 (01-Mar-2017)
    • Upgraded to SSH/J 0.20.0.
    • Added suPassword to hidden fields of ConnectionOptions to prevent printing it.
  • Overthere 4.4.3 (18-Jan-2017)
    • Added missing PKCS5 KeyFileProvider.
    • Fixed hanging Tunnel connection issue.
    • Upgraded to SSH/J 0.19.1.
    • Upgraded to SMB/J 0.0.7.
  • Overthere 4.4.2 (12-Dec-2016)
    • Avoided creation of SecureRandom for every connection, reducing the amount of entropy needed.
    • Upgraded to SSH/J 0.19.0.
    • Upgraded to Scannit v1.4.1 (includes fix for #179 and #183).
    • Fixed #192.
  • Overthere 4.4.1 (07-Oct-2016)
    • Fixed bug causing incorrect parent path resolution for SmbFile.
  • Overthere 4.4.0 (22-Sep-2016)
    • Added SMB protocol support #186.
  • Overthere 4.2.2 (12-Jul-2016)
  • Overthere 4.3.3 (8-Jul-2016)
    • Fixed #172, #176 and #182.
    • Fixed bug where execution of commands on WINRM_NATIVE connections would fail if an argument contained special characters.
    • Upgraded to SSH/J 0.17.2 (includes fix for SSH/J issue #252).
  • Overthere 4.3.2 (12-Apr-2016)
    • Fixed execution of some commands on localhost when the OS is Windows.
    • Upgraded to SSH/J 0.16.0.
  • Overthere 4.3.1 (01-Apr-2016)
    • Fixed ClassCastException that occurred when passing an SshConnectionType as a string.
  • Overthere 4.3.0 (15-Mar-2016)
  • Overthere 4.1.2 (23-Oct-2015)
    • NOTE: This release contains the same code as 4.2.1, except for all the library upgrades of 4.2.0
    • Fixed #165: Fixed slowness of SFTP copy operation(s).
  • Overthere 4.2.1 (21-Oct-2015)
    • Fixed #165: Fixed slowness of SFTP copy operation(s).
  • Overthere 4.2.0 (06-Oct-2015)
    • Upgraded to Java 7.
    • Added more debug logging when closing streams.
    • Fix typo in logging of WsmanSPNegoSchemeFactory
    • Upgraded to scannit 1.4.0, fixes Java8 compatibility.
    • Upgraded to bouncy castle 1.52.
    • Upgraded to commons-net 3.3.
    • Upgraded to commons-codec 1.10.
    • Upgraded to slf4j 1.7.12.
  • Overthere 4.1.1 (26-Aug-2015)
    • Fixed bug where WINRM_INTERNAL connection with Windows domain account failed with error message "Unexpected HTTP response on http://hostname/wsman: (401)".
  • Overthere 4.1.0 (18-Aug-2015)
    • Fixed bug where temporary files were not deleted for a CIFS/WINRM_INTERNAL or a CIFS/TELNET connection when connection was closed.
    • Fixed bug where short-circuit copy would fail with error message "tar: can't set time on .: Not owner". This could occur when copying from/to a temporary file/directory for an SSH/SU, SSH/SUDO or SSH/INTERACTIVE_SUDO connection.
    • Fixed bug where statting a file would fail with error message "ls -ld /path/to/dir returned 0 but its output is unparseable: ESC" for an SSH/SCP, SSH/SU, SSH/SUDO or SSH/INTERACTIVE_SUDO connection.
    • Added logging (at DEBUG level) for local file operations.
    • Disconnected from local: messages are now logged at DEBUG level instead of INFO level.
    • Upgraded to SSH/J 0.13.0.
  • Overthere 4.0.1 (02-Jun-2015)
    • Implemented correct fix for #153.
  • Overthere 4.0.0 (17-Apr-2015)
    • The binary incompatibility introduced in Overthere 2.4.7 and present in 2.4.8 and 3.0.0 was breaking too many libraries and too much code using Overthere. We've reverted the breaking change so that LocalFile.valueOf once again returns an OverthereFile and added a new LocalFile.from method which returns a LocalFile. Because this version is binary incompatible with 3.0.0 we had to bump the major version again but this is one is binary compatible with Overthere 2.4.6 and below.
  • Overthere 3.0.0 (14-Apr-2015)
    • Please do not use this version of Overthere as it is not binary compatible with Overthere 2.4.6 and lower. Please use Overthere 4.0.0 and up.
    • Because of the binary incompatiblity introduced in Overthere 2.4.7, we've decided to bump the major version and release this version of Overthere 3.0.0.
    • Upgraded SSHJ to 0.12.0.
    • Can now configure socket timeout, fixes #156 and #158
    • LocalConnection does not log 'Connection [LOCAL:] was not closed, closing automatically.'
  • Overthere 2.4.8 (05-Feb-2015)
    • Please do not use this version of Overthere as it is not binary compatible with Overthere 2.4.6 and lower. Please use Overthere 4.0.0 and up.
    • Upgraded Scannit to 1.3.1.
    • Checking whether the CifsFile.getParentFile() is valid, fixes #153
    • Fixed #145.
  • Overthere 2.4.7 (02-Feb-2015)
    • Please do not use this version of Overthere as it is not binary compatible with Overthere 2.4.6 and lower. Please use Overthere 4.0.0 and up.
    • Upgraded SSHJ to 0.11.0.
    • Changed return type of LocalFile.valueOf(java.io.File) from OverthereFile to LocalFile. This breaks binary compatibility with Overthere 2.4.6.
    • Fixed #139 and #146.
  • Overthere 2.4.6 (09-Jan-2015)
    • Upgraded Scannit to 1.3.0 to remove transitive Guava dependency.
    • Upgraded Gradle to 2.2.1.
    • Fixed #133, #134 and #138.
  • Overthere 2.4.5 (27-Oct-2014)
  • Overthere 2.4.4 (27-Aug-2014)
  • Overthere 2.4.3 (17-May-2014)
  • Overthere 2.4.2 (26-Mar-2014)
    • Bumped Guava to version 16.0.1. This ensures that overthere should work on JDK 1.7.0_51 and up.
  • Overthere 2.4.1 (24-Mar-2014)
    • Fixed race condition in creation of temporary directories.
  • Overthere 2.4.0 (12-Mar-2014)
    • Added support for the the SU connection type, which fixes #102, and reverted the fix for #89.
    • Improved efficiency of copy operations on remote hosts by using a copy command on that remote host instead of downloading and then uploading the file or directory, which fixes #91. Note that this behaviour is only invoked when copying files or directories on a remote host, not when copying them between remote hosts.
    • Fixed #87, #88, #96, #99, #103, #104.
  • Overthere 2.3.1 (16-Jan-2014)
    • Fixed #89
    • Fixed race condition in creation of local temporary directories.
  • Overthere 2.3.0 (25-Oct-2013)
    • Implemented support for winrs, the native WinRM implementation available on Windows hosts, which fixes [#12|#12]. N.B.: To distinguish this connection type from the existing Java one, the connection type WINRM has been replaced by WINRM_INTERNAL (the Java implementation) and WINRM_NATIVE (the Windows implementation).
    • Added refreshKrb5Config=true option to the Kerberos JAAS configuration to make sure the configuration is re-read for every request.
    • Upgraded the SSH/J 0.9.0.
    • Fixed bug that occurred when reading or writing many files over a single SFTP connection to a WinSSHD server.
  • Overthere 2.2.2 (28-Aug-2013)
  • Overthere 2.2.1 (24-Jul-2013)
    • Fully implemented OverthereConnection.startProcess() for CIFS/WinRM connections, which fixes #54 properly. The previous implementation did not handle stdin.
    • Fixed #57, #72, #73, #76, #77 and #79.
    • Updated documentation and troubleshooting guides for SSH, CIFS, WinRM and Kerberos.
    • Some minor code and documentation fixes.
  • Overthere 2.2.0 (07-Feb-2013)
    • Introduced OverthereExecutionOutputHandler interface to allow stderr to also be captured character by character, which fixes #67.
    • Made the commands used for SSH/SCP, SSH/SUDO and SSH/INTERACTIVE_SUDO connections configurable, which fixes #52.
    • Made canStartProcess() return false for CIFS/WinRM connections because its startProcess implementation does not correctly handle stdin, disables fix for issue #54.
    • Fixed #65, #68 and #70.
    • Some minor code fixes.
    • Some not-so-minor documentation improvements: more setup and troubleshooting info for WinRM and Kerberos.
  • Overthere 2.1.1 (17-Dec-2012)
  • Overthere 2.1.0 (26-Oct-2012)
    • Re-enabled support for Windows domain accounts in CIFS/Telnet connections, which fixes #60.
    • Fixed Kerberos code to use WSMAN SPN by default, which fixes #58.
    • Added option to add the port to the SPN, which fixes #49.
    • Added support for Negotiate authentication (Kerberos only), which fixes #59.
    • Some minor code and documentation fixes.
  • Overthere 2.1.0-beta-1 (21-Sep-2012)
    • Implemented OverthereProcess.startProcess() for CIFS/WinRM connections, which fixes #54.
    • Fixed #53 and #55.
    • Some minor code and documentation fixes.
  • Overthere 2.0.0 (22-Aug-2012)
    • Stable release of Overthere 2.0.0.
    • Some minor code and documentation fixes.
  • Overthere 2.0.0-rc-1 (17-Aug-2012)
  • Overthere 2.0.0-beta-7 (02-Aug-2012)
    • Fixed bug in WinRM implementation: It was not sending individual stdout chars to OverthereProcessOutputHandler.handleOutput().
  • Overthere 2.0.0-beta-6 (02-Aug-2012)
    • Renamed CIFS_PATH_SHARE_MAPPING back to PATH_SHARE_MAPPINGS.
  • Overthere 2.0.0-beta-5 (02-Aug-2012)
    • Added support for Windows domain accounts to CIFS and WinRM connection methods.
    • Renamed a few options.
    • Fixed bug in SSH tunnel port allocation code that caused the same local port to be allocated multiple times on Windows.
    • Changed license to GPLv2 with XebiaLabs FLOSS License Exception.
  • Overthere 2.0.0-beta-4 (19-Jun-2012)
    • Fixed #42.
    • Moved the itest-support sub project out to new Github repository Overcast
    • Updated documentation.
  • Overthere 2.0.0-beta-3 (27-Mar-2012)
    • Updated documentation.
  • Overthere 2.0.0-beta-2 (23-Mar-2012)
    • Fixed #39 and #40.
    • Upgraded to latest jCIFS to fix issues with windows domain names and stability using tunnels.
    • Set default pty to true in case of interactive sudo and no pty set.
  • Overthere 1.0.17 (20-Mar-2012)
  • Overthere 2.0.0-beta-1 (05-Mar-2012)
    • Re-implemented SSH tunnels. Tunnels are now created on demand instead of the user having to specify the localPortForwards explicitly. This makes management of tunnels easier and prevents clashes.
    • Ported Overthere tests to use TestNG instead of JUnit.
  • Overthere 1.0.16 (23-Feb-2012)
    • Reverted changes made to support SSH tunnels in 1.0.14 and 1.0.15 because it did not work as well as we hoped. We are reimplementing it for Overthere 2.0 to be released early March.
    • Fixed command line encoding bugs for SSH/CYGWIN on Windows:
      • Now transforming the first element of the command line to a Cygwin path so that batch files (and executables) in specific directories (instead of on the PATH) can be executed.
      • Encoding the command line as if the target OS is UNIX because OpenSSH on Cygwin uses Unix encoding.
  • Overthere 1.0.15 (21-Feb-2012)
    • Added explicit close() method to the new com.xebialabs.overthere.OverthereConnection interface (it was a class in 1.0.13) that does not throw java.io.IOException.
  • Overthere 1.0.14 (20-Feb-2012)
    • Added support for SSH tunnels to jumpstations.
    • Added support for NTLM authentication.
    • Upgraded to SSH/J 0.7.0.
  • Overthere 1.0.13 (18-Jan-2012)
    • Masked passwords in logging.
    • Made ItestHostFactory also look for itest.properties in ~/.itest (in addition to the classpath and the current working directory).
  • Overthere 1.0.12 (12-Jan-2012)
    • Allowed forward slashes (/) to be used in Windows paths.
    • Made it possible to access non-administrative shares on Windows so that the CIFS connection methods can be used with regular user accounts. See the pathShareMappings connection option.
    • Added the allocatePty connection option to specify an explicit pseudo terminal to use.
  • Overthere 1.0.11 (09-Dec-2011)
    • Fixes to the SSH/WinSSHD implementation and a few other minor fixes.
    • Added a lot of documentation.
    • Added examples project.
    • Changed license to ASLv2.
  • Overthere 1.0.10 (23-Nov-2011)
    • Added support for SSH/WinSSHD on Windows.
  • Overthere 1.0.9 (22-Nov-2011)
    • Initial public release with support for SSH on Unix as well as CIFS/TELNET, CIFS/WinRM and SSH/CYGWIN on Windows.

About

Runs something "Over there"

www.xebialabs.com

Resources

Readme

License

View license
Activity
Custom properties

Stars

196 stars

Watchers

66 watching

Forks

66 forks
Report repository

Releases 1

Overthere 2.4.3 Latest
May 23, 2014

Packages

No packages published

Contributors 56

+ 42 contributors

Languages