Skip to content

Admin

Jump to bottom
tiedemann edited this page Jul 9, 2018 · 4 revisions

Install OPUS-related Software for NLPL

Build scripts are available in directory $NLPL_HOME/software/build_scripts. Install scripts and modules by running

make -C software/build_scripts list
make -C software/build_scripts install

All software packages (by running all Makefiles in all sub-dirs of build_scripts) can also be installed with the same Makefile:

source software/build_scripts/clean-env.sh
make -C software/build_scripts install-software

Software and modules will be installed in $NLPL_HOME/software and $NLPL_HOME/modules.

Configuration of the OPUS server via cPouta

Basic steps for preparing to use cloud services:

Basic steps for launching cPouta virtual machines (steps 3.2.2 and 3.2.3):

Basic steps for attaching persistent storage

Setup the new volume on the instance (first time only). See https://research.csc.fi/pouta-persistent-volumes Assume we have volumes for OPUS, mysql, cwb and letsmt. Doing it for OPUS (assuming /dev/vdb)

sudo parted -l
sudo parted -a optimal /dev/vdb mklabel gpt
sudo parted -a optimal /dev/vdb mkpart primary 0% 100%
sudo mkfs.ext4 /dev/vdb1
sudo e2label /dev/vdb1 OPUS
sudo mkdir /media/OPUS
sudo mount /dev/vdb1 /media/OPUS

OS setup and configuration

ubuntu packages to be installed (from old opus server)

cut -f1  ubuntu-packages-January.txt | grep -v linux | grep -v amd | grep -v x11 | grep -v xserver | grep -v xorg | grep -v xin | grep -v xterm | grep -v gnome | grep -v tex | grep -v nautilus | grep -v abiword | grep -v font | grep -v gir | grep -v cups | grep -v grub | grep -v ghost | grep -v grid | grep -v notepad | tr "\n" ' '

gives this list (do we need that all?):

accountsservice acl acpi-support acpid adduser alsa-base alsa-utils anacron apache2 apache2-bin apache2-data apache2-mpm-prefork apache2-utils apg app-install-data apparmor apport apport-gtk apport-symptoms apt apt-show-versions apt-transport-https apt-utils apt-xapian-index aptdaemon aptdaemon-data aptitude aptitude-common apturl apturl-common aspell aspell-en at at-spi2-core audacious audacious-plugins-data autoconf automake autotools-dev avahi-daemon avahi-utils awstats base-files base-passwd bash bash-completion bc bind9-host binutils biosdevname blueman bluez bsdmainutils bsdutils build-essential busybox-initramfs busybox-static byobu bzip2 ca-certificates ca-certificates-java cheese-common cmake cmake-data colord command-not-found command-not-found-data console-setup consolekit coreutils cpio cpp cpp-4.8 cracklib-runtime crda cron curl cython cython3 dash dbus dc dconf-cli dconf-service debconf debconf-i18n debianutils default-jre default-jre-headless desktop-base desktop-file-utils dh-python dialog dictionaries-common diffstat diffutils discover discover-data dmidecode dmsetup dmz-cursor-theme dnsmasq-base dnsutils docbook-xml docutils-common docutils-doc dosfstools doxygen dpkg dpkg-dev e2fsprogs ed eject emacs emacs24 emacs24-bin-common emacs24-common emacs24-common-non-dfsg emacsen-common enchant ethtool evince evince-common evolution-data-server evolution-data-server-common evolution-data-server-online-accounts fakeroot ffmpegthumbnailer file file-roller findutils firefox firefox-locale-en foomatic-db-compressed-ppds friendly-recovery ftp fuse g++ g++-4.8 galculator gawk gcc gcc-4.8 gconf-service gconf-service-backend gconf2 gconf2-common gcr gdb gdebi gdebi-core gdisk gecko-mediaplayer genisoimage geoclue geoclue-ubuntu-geoip geoip-database git git-man gkbd-capplet gksu glib-networking-common glib-networking-services gnumeric gnumeric-common gnumeric-doc gnupg gpgv gpicview graphviz grep groff-base gsettings-desktop-schemas gsettings-ubuntu-schemas gstreamer1.0-clutter gucharmap guile-2.0-libs guvcview gvfs-backends gvfs-common gvfs-daemons gvfs-fuse gzip hardening-includes hardinfo hdparm hicolor-icon-theme hostname hplip hplip-data htop humanity-icon-theme hunspell-en-ca hunspell-en-us hwdata ibus icedtea-netx-common ifupdown im-config imagemagick-common indicator-application indicator-application-gtk2 indicator-bluetooth indicator-datetime indicator-keyboard indicator-power indicator-session indicator-sound info init-system-helpers initramfs-tools initramfs-tools-bin initscripts inputattach insserv install-info installation-report intltool-debian iproute2 iptables iputils-arping iputils-ping iputils-tracepath irqbalance isc-dhcp-client isc-dhcp-common iso-codes java-common javascript-common kbd kerneloops-daemon keyboard-configuration klibc-utils kmod krb5-locales landscape-common language-pack-en language-pack-en-base language-selector-common laptop-detect leafpad less liba52-0.7.4 libaccount-plugin-1.0-0 libaccount-plugin-generic-oauth libaccount-plugin-google libaccounts-qt5-1 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libapache2-mod-apreq2 libapache2-mod-perl2 libapache2-mod-php5 libapache2-mod-python libapache2-reload-perl libapache2-request-perl libapparmor-perl libappindicator3-1 libapreq2-3 libapt-pkg-perl libarchive-extract-perl libarchive-zip-perl libasound2-data libaspell15 libatk-wrapper-java libatk1.0-data libaudit-common libauthen-sasl-perl libautodie-perl libbind9-90 libblas3 libbonobo2-common libbs2b0 libbsd-resource-perl libburn4 libc-bin libc-dev-bin libcairo-perl libcamel-1.2-45 libcap-ng0 libcap2-bin libcddb2 libcdio-cdda1 libcdio-paranoia1 libcdio13 libcdt5 libcgraph6 libclass-accessor-perl libclass-inspector-perl libclass-load-perl libclone-perl libclutter-1.0-common libcogl-common libcommon-sense-perl libcompfaceg1 libcrypt-passwdmd5-perl libcrypt-ssleay-perl libcue1 libcwidget3 libdaemon0 libdata-optlist-perl libdbd-mysql-perl libdbi-perl libdevel-symdump-perl libdigest-hmac-perl libdiscover2 libdist-checkconflicts-perl libdns100 libdpkg-perl libdrmaa-java libebackend-1.2-7 libebook-1.2-14 libebook-contacts-1.2-0 libecal-1.2-16 libedata-book-1.2-20 libedata-cal-1.2-23 libedataserver-1.2-18 libemail-valid-perl libencode-locale-perl liberror-perl libestr0 libevdocument3-4 libevview3-3 libexo-common libexo-helpers libffmpegthumbnailer4 libfile-basedir-perl libfile-bom-perl libfile-copy-recursive-perl libfile-desktopentry-perl libfile-fcntllock-perl libfile-listing-perl libfile-mimeinfo-perl libfile-remove-perl libfile-slurp-perl libfm-data libfm-extra4 libfm-gtk-data libfm-gtk4 libfm-modules libfm4 libfuse-dev libgcr-3-common libgda-5.0-4 libgda-5.0-common libgdata-common libgdata13 libgdk-pixbuf2.0-common libgdome2-0 libgdome2-cpp-smart0c2a libgksu2-0 libglib-perl libglib2.0-bin libglib2.0-data libgmtk1-data libgoa-1.0-common libgoffice-0.10-10 libgoffice-0.10-10-common libgphoto2-l10n libgpod-common libgrail6 libgrip0 libgs9 libgs9-common libgsasl7 libgsf-1-114 libgsf-1-common libgssdp-1.0-3 libgtk-3-bin libgtk-3-common libgtk2-perl libgtk2.0-bin libgtk2.0-common libgtkmathview0c2a libgtkspell0 libgtop2-7 libgtop2-common libgucharmap-2-90-7 libgupnp-1.0-4 libgutenprint2 libgvc6 libgvpr2 libgweather-3-6 libgweather-common libhook-lexwrap-perl libhpmud0 libhtml-form-perl libhtml-format-perl libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhtml-tree-perl libhttp-cookies-perl libhttp-daemon-perl libhttp-date-perl libhttp-message-perl libhttp-negotiate-perl libical1 libid3tag0 libidl-common libijs-0.35 libimlib2 libindicator3-7 libindicator7 libinline-files-perl libinline-perl libio-html-perl libio-pty-perl libio-socket-inet6-perl libio-socket-ssl-perl libio-string-perl libipc-run-perl libipc-system-simple-perl libisc95 libisccc90 libisccfg90 libisofs6 libjbig2dec0 libjs-excanvas libjs-jquery libjs-jquery-timepicker libjs-jquery-ui libjson-glib-1.0-common libjson-perl libjson-xs-perl libjte1 libklibc libkpathsea6 liblapack3 liblightdm-gobject-1-0 liblink-grammar4 liblircclient0 liblist-moreutils-perl liblockfile-bin liblog-dispatch-perl liblog-log4perl-perl liblog-message-simple-perl libloudmouth1-0 liblwp-mediatypes-perl liblwp-protocol-https-perl liblwres90 libm17n-0 libmailtools-perl libmailutils4 libmeanwhile1 libmenu-cache-bin libmenu-cache3 libmessaging-menu0 libminiupnpc8 libmodplug1 libmodule-implementation-perl libmodule-install-perl libmodule-pluggable-perl libmodule-runtime-perl libmodule-scandeps-perl libmtp-common libmtp-runtime libmusicbrainz3-6 libnatpmp1 libneon27-gnutls libnet-dbus-perl libnet-dns-perl libnet-domain-tld-perl libnet-http-perl libnet-ip-perl libnet-smtp-ssl-perl libnet-ssleay-perl libnet-xwhois-perl libnm-glib-vpn1 libnm-glib4 libnm-gtk-common libnm-gtk0 libnm-util2 libnotify-bin libnss3-nssdb libobrender29 libobt2 libonig2 liboobs-1-5 libopenobex1 libopus0 libots0 libpackage-stash-perl libpackage-stash-xs-perl libpam-modules-bin libpam-runtime libpango-perl libpaper-utils libpar-dist-perl libparams-classify-perl libparams-util-perl libparams-validate-perl libparse-debianchangelog-perl libparse-recdescent-perl libpathplan4 libperl5.18 libperlio-gzip-perl libpisock9 libpostproc52 libppi-perl libpurple-bin libpurple0 libpwquality-common libquvi-scripts librarian0 libreadonly-perl libreadonly-xs-perl libruby1.9.1 libsane-common libsane-hpaio libsecret-common libsemanage-common libsepol1-dev libsgutils2-2 libsignon-extension1 libsignon-glib1 libsignon-plugins-common1 libsignon-qt5-1 libsnmp-base libsocket6-perl libssl-doc libsub-identify-perl libsub-install-perl libsub-name-perl libsub-uplevel-perl libt1-5 libtask-weaken-perl libterm-readkey-perl libterm-ui-perl libtest-differences-perl libtest-nowarnings-perl libtest-object-perl libtest-requires-perl libtest-subcalls-perl libtest-tester-perl libtest-use-ok-perl libtest-warn-perl libthai-data libtidy-0.99-0 libtie-ixhash-perl libtimedate-perl libtimezonemap1 libtool libtree-dagnode-perl libtry-tiny-perl libuniconf4.6 libunity-control-center1 libunity-scopes-json-def-desktop liburi-perl liburi-query-perl libusbmuxd2 libutempter0 libvariable-magic-perl libvte-2.90-9 libvte-2.90-common libvte-common libvte9 libwacom-common libwebcam0 libwebkitgtk-3.0-common libwhoopsie0 libwnck-3-common libwnck-common libwnck22 libwpd-0.9-9 libwpg-0.2-2 libwps-0.2-2 libwvstreams4.6-base libwvstreams4.6-extras libwww-perl libwww-robotrules-perl libxapian22 libxfce4ui-1-0 libxfce4ui-common libxfce4util-bin libxfce4util-common libxfce4util6 libxfconf-0-2 libxklavier16 libxml-libxml-perl libxml-namespacesupport-perl libxml-parser-perl libxml-perl libxml-sax-base-perl libxml-sax-expat-perl libxml-sax-perl libxml-simple-perl libxml-twig-perl libxml-writer-perl libxml-xpathengine-perl libxml2-utils libxtables10 libyaml-libyaml-perl libyaml-perl libyaml-tiny-perl libyelp0 light-locker light-locker-settings lightdm lightdm-gtk-greeter link-grammar-dictionaries-en links lintian lksctp-tools lm-sensors lmodern locales lockfile-progs login logrotate lp-solve lsb-base lsb-release lshw lsof ltrace lubuntu-artwork lubuntu-artwork-14-04 lubuntu-core lubuntu-default-session lubuntu-default-settings lubuntu-desktop lubuntu-icon-theme lubuntu-lxpanel-icons lubuntu-software-center lxappearance lxappearance-obconf lxlauncher lxmenu-data lxpanel lxpanel-indicator-applet-plugin lxrandr lxsession lxsession-data lxsession-default-apps lxsession-logout lxshortcut lxtask lzma-dev m17n-contrib m17n-db m4 mailutils mailutils-common make makedev man-db manpages manpages-dev mawk mc mc-data memtest86+ mime-support mlocate mobile-broadband-provider-info modemmanager module-init-tools mount mountall mousetweaks mplayer2 mscompress mtools mtpaint mtr-tiny multiarch-support mysql-client-5.5 mysql-client-core-5.5 mysql-common mysql-server mysql-server-5.5 mysql-server-core-5.5 nano ncurses-base ncurses-bin ncurses-term net-tools netbase netcat-openbsd network-manager network-manager-pptp ntfs-3g ntp ntpdate obconf obex-data-server obexd-client openbox openjdk-6-jre-lib openprinting-ppds openssh-client openssh-server openssh-sftp-server openssl os-prober p11-kit packagekit packagekit-backend-aptcc packagekit-tools parted passwd patch patchutils pciutils pcmanfm pcmciautils perl perl-base perl-doc perl-modules php5-cli php5-common php5-json php5-mysql php5-readline pidgin pidgin-data pidgin-libnotify pkg-config plymouth plymouth-label plymouth-theme-lubuntu-logo pm-utils policykit-1 policykit-desktop-privileges poppler-data poppler-utils popularity-contest postfix powermgmt-base ppp pppconfig pppoeconf printer-driver-c2esp printer-driver-foo2zjs printer-driver-foo2zjs-common printer-driver-gutenprint printer-driver-min12xxw printer-driver-pnm2ppa printer-driver-postscript-hp printer-driver-ptouch printer-driver-pxljr printer-driver-sag-gdi printer-driver-splix procps psmisc pulseaudio pulseaudio-utils python python-apt python-apt-common python-aptdaemon python-aptdaemon.gtk3widgets python-babel python-babel-localedata python-cairo python-chardet python-colorama python-commandnotfound python-configobj python-crypto python-dbus python-dbus-dev python-debian python-defer python-dev python-distlib python-docutils python-gconf python-gdbm python-genshi python-gi python-gobject python-gobject-2 python-gtk2 python-gudev python-html5lib python-imaging python-ldb python-libxml2 python-minimal python-notify python-openssl python-pam python-pexpect python-pil python-pip python-pkg-resources python-psutil python-pycurl python-pygments python-pysqlite2 python-renderpm python-reportlab python-reportlab-accel python-requests python-roman python-samba python-serial python-setuptools python-six python-smbc python-subversion python-talloc python-tdb python-twisted-bin python-twisted-core python-tz python-urllib3 python-wheel python-xapian python-xdg python-zope.interface python2.7 python2.7-dev python2.7-minimal python3 python3-apport python3-apt python3-aptdaemon python3-aptdaemon.gtk3widgets python3-chardet python3-commandnotfound python3-dbus python3-debian python3-defer python3-dev python3-distupgrade python3-gi python3-minimal python3-newt python3-numpy python3-packagekit python3-pkg-resources python3-problem-report python3-pycurl python3-six python3-software-properties python3-update-manager python3-xkit python3.4 python3.4-dev python3.4-minimal qpdf rarian-compat readline-common recode resolvconf rfkill rsync rsyslog rtkit ruby ruby1.9.1 run-one samba-common samba-common-bin sane-utils screen scrot sed sensible-utils session-migration sessioninstaller sgml-base sgml-data shared-mime-info signon-keyring-extension signon-plugin-oauth2 signon-ui signond simple-scan smbclient software-properties-common software-properties-gtk sound-theme-freedesktop ssh-import-id ssl-cert strace subversion sudo swig swig2.0 sylpheed sylpheed-doc sylpheed-i18n sylpheed-plugins synaptic system-config-printer-common system-config-printer-udev system-tools-backends systemd-services systemd-shim sysv-rc sysvinit-utils t1utils tar tasksel tasksel-data tcl tcl8.6 tcpd tcpdump tcsh telnet tidy time tk tk8.6 tmux toshset trac transmission transmission-common transmission-gtk tsconf ttf-dejavu-extra tzdata tzdata-java ubuntu-drivers-common ubuntu-extras-keyring ubuntu-keyring ubuntu-minimal ubuntu-release-upgrader-core ubuntu-release-upgrader-gtk ubuntu-standard ubuntu-system-service ucf udev udisks2 ufw unattended-upgrades unison unity-control-center unity-control-center-signon unity-greeter unity-settings-daemon unzip update-inetd update-manager update-manager-core update-notifier update-notifier-common upower upstart ureadahead usb-creator-common usb-creator-gtk usb-modeswitch usb-modeswitch-data usbmuxd usbutils uuid-runtime uvcdynctrl uvcdynctrl-data vbetool vim vim-common vim-runtime vim-tiny w3m wamerican wbritish wget whiptail whoopsie wireless-regdb wireless-tools wpasupplicant wvdial xauth xbitmaps xdg-user-dirs xdg-user-dirs-gtk xdg-utils xfburn xfce4-notifyd xfce4-power-manager xfce4-power-manager-data xfconf xkb-data xml-core xpad xsltproc xtrans-dev xul-ext-ubufox xz-utils yelp yelp-xsl zenity zenity-common zip

Apache configuration

Check and adjust all configuration files in /etc/apache2

  • admin/apache2/ includes configurations from old OPUS server
  • enable all mods
  • adjust opus.conf and enable site (disable default): change all /home/opus/ to /media/OPUS/ and /home/letsmt/ to /media/letsmt
  • restart apache server with apache2ctl restart

There is no DNS service from CSC (see https://research.csc.fi/pouta-dns)

Setup website and Trac (Wiki for OPUS and LetsMT!)

  • mount letsmt volume (assume it's /dev/vdd)
sudo parted -l
sudo parted -a optimal /dev/vdd mklabel gpt
sudo parted -a optimal /dev/vdd mkpart primary 0% 100%
sudo mkfs.ext4 /dev/vdd1
sudo e2label /dev/vdd1 letsmt
sudo mkdir -p /media/letsmt
sudo mount /dev/vdd1 /media/letsmt
  • prepare permissions
sudo chown cloud-user /media/OPUS /media/letsmt
sudo chmod 755 /media/OPUS /media/letsmt
  • fetch the www-data from opus
rsync --exclude '*.tar.gz' -zav [email protected]:/home/opus/public_html /media/OPUS/
rsync --exclude '*.tar.gz' -zav [email protected]:/home/letsmt/public_html /media/letsmt/
rsync -zav [email protected]:/home/opus/trac /media/OPUS/
rsync -zav [email protected]:/home/letsmt/trac /media/letsmt
  • create some symbolic links (for compatibility)
sudo ln -s /media/OPUS /home/opus
sudo ln -s /media/letsmt /home/letsmt
  • get the corpus databases
mkdir /media/OPUS/html
rsync -zav [email protected]:/home/opus/OPUS/html/*.db /media/OPUS/html/

Setup of the MySQL WordAlign database

  • NEW PLACE ON CURRENT SERVER: /media/OPUS/mysql

The instructions below apply to a creating a separate Volume for the database! This is not necessary if we place it in the OPUS Volume! Below are the old instructions:

  • create a volume for MySQL
  • prepare the volume for mounting (first time only), assuming it is vdc 
sudo parted -l
sudo parted -a optimal /dev/vdc mklabel gpt
sudo parted -a optimal /dev/vdc mkpart primary 0% 100%
sudo mkfs.ext4 /dev/vdc1
sudo e2label /dev/vdc1 mysql
sudo mkdir /media/mysql
sudo mount /dev/vdc1 /media/mysql

stop MySQL and move to new volume

sudo /etc/init.d/mysql stop
sudo mv /var/lib/mysql /media/mysql/
sudo mv /var/lib/mysql-files /media/mysql/
sudo cp /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/usr.sbin.mysqld.BACKUP
sudo sed 's#/var/lib/#/media/mysql/#' < /etc/apparmor.d/usr.sbin.mysqld.BACKUP > /etc/apparmor.d/usr.sbin.mysqld
sudo cp /etc/mysql/my.cnf /etc/mysql/my.cnf.BACKUP
sudo sed 's#/var/lib/#/media/mysql/#' < /etc/mysql/my.cnf.BACKUP > /etc/mysql/my.cnf

sync databases from taito

sudo rsync -zav [email protected]:/proj/OPUS/admin/mysql/mysql /media/mysql/mysql/
sudo rsync -zav [email protected]:/proj/OPUS/admin/mysql/opus /media/mysql/mysql/
sudo chown -R mysql:mysql /media/mysql
sudo /etc/init.d/mysql start

NEW: mySQL is now in /media/OPUS/mysql

PostgreSQL

PostgreSQL is used by multilingwis2

  • compile & install postgres > 9.6 from source (see ~/admin/postgres)
  • create a data directory
  • initialise and start postgresql
sudo mkdir -p /media/OPUS/pgsql/data
sudo adduser postgres
sudo chown -R postgres /media/OPUS/pgsql
sudo ln -s /usr/local/pgsql/bin/* /usr/local/bin/
sudo ln -s /usr/local/pgsql/lib/* /usr/local/lib/
sudo ln -s /usr/local/pgsql/share/* /usr/local/share/
su - postgres
/usr/local/pgsql/bin/initdb -D /media/OPUS/pgsql/data
/usr/local/pgsql/bin/postgres -D /media/OPUS/pgsql/data >/media/OPUS/pgsql/postgresql.log 2>&1 &

Web statistics

  • awstats is configured in /etc/awstats/awstats.conf
Include "/etc/awstats/awstats.conf.local"

LogFile="/var/log/apache2/access.log"
SiteDomain="opus.nlpl.eu"
HostAliases="localhost 127.0.0.1 opus.nlpl.eu"
  • configuration for apache2 (/etc/apache2/sites-available/opus.conf)
Alias /awstatsclasses "/usr/share/awstats/lib/"
Alias /awstats-icon/ "/usr/share/awstats/icon/"
Alias /awstatscss "/usr/share/doc/awstats/examples/css"
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
ScriptAlias /statistics/ /usr/lib/cgi-bin/
Options +ExecCGI +SymLinksIfOwnerMatch
  • script is in /usr/lib/cgi-bin/awstats.pl linked to http://opus.nlpl.eu/statistics/awstats.pl
  • data dir is /var/lib/awstats/ (make sure it's writable for www-data)
  • make sure that log files are readable for www-data, change log rotate for apache (/etc/logrotate.d/apache2) from "create 640 root adm" to "create 640 www-data adm"
  • make sure that awstats cache dir is writable for www-data (/var/cache/awstats)
  • cron job in /etc/cron.d/awstats
MAILTO=root

*/10 * * * * www-data [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh

# Generate static reports:
10 03 * * * www-data [ -x /usr/share/awstats/tools/buildstatic.sh ] && /usr/share/awstats/tools/buildstatic.sh

Other admin tasks

  • resize persistent volumes

If you use partitions on the block device, you will need to grow the size of the partition before you grow the size of the filesystem.

Generic instructions: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/s2-disk-storage-parted-resize-part.html

Commands something like:

parted
resize 1 oldmbs newmbs

After that is done and written, you should be able to run resize2fs on /dev/vdb1.

Using Object Storage on Taito

You can use Object Storage on Taito with Swift client once you have configured the environment variables.

To configure your environment variables, download the script from:

https://pouta.csc.fi/dashboard/project/access_and_security/api_access/openrc/

and run it:

source project_<project-id>-openrc.sh

You can now run Swift client, for example, list all containers:

swift list

or list all objects in a container:

swift list <container-name>

Upload objects to a container:

swift upload <container-name> <object-name>

If a container named <container-name> does not exist, one will be created. Containers are private by default. You can set them public by defining the metadata:

swift post <container-name> --read-acl ".r:*"

Objects larger than 5G need to be segmented when uploaded:

swift upload <container-name> --use-slo --segment-size 5G <object-name>

5G is the biggest segment size allowed, using this will create smallest number of individual objects to store. You can also upload objects smaller than 5G with the segmenting command, those will not be segmented.

Whole directories can also be uploaded. For example, to upload directory Europarl to container OPUS:

swift upload OPUS --use-slo --segment-size 5G Europarl

The previous command uses segmenting in case there are individual files larger than 5G.

Delete objects with:

swift delete <container-name> <object-name>

Download objects from URLs with format: "https://object.pouta.csc.fi/container-name/object-name"

For example: https://object.pouta.csc.fi/OPUS/Europarl/v7/xml/en-fi.xml.gz

F.A.Q.

  • Can I run cronjobs on taito? Yes it's possible. One needs to remember which server it's installed, so please use taito-login3.csc.fi or taito-login4.csc.fi instead of generic taito.csc.fi alias.
  • Can I run cronjobs on abel? Yes. Set them from the login nodes with crontab -e
  • How do I use the cPouta service on CSC? check the user guide at https://research.csc.fi/pouta-user-guide
  • Can I create a disk image of my virtual server on CSC? Yes! Check https://research.csc.fi/pouta-adding-images
  • What's about DNS? Check https://research.csc.fi/pouta-dns
  • How to find out the DNS name of the floating IP: use dig -x ip-address
Clone this wiki locally