git - Ubuntu-Server: Every SSH-Connect creates non deleted cgroup -


we running ubuntu 14.04 servers somehow create new cgroup each ssh-session created, sadly cgroups not deleted when session closed.

as running gitlab on 1 of servers , git-clients poll once per minute end thousands of zombie cgroups git-user per week.

about once per month reach cgroup-limit leading problems docker saying:

starting container process caused "process_linux.go:258: applying cgroup configuration process caused \"mkdir /sys/fs/cgroup/memory/docker/28485427c2689d7a2c4be83990af873e9ab5732c5c741f0d10bfc3ba21b1d167: no space left on device\""

it seems problem occurs on servers having docker installed. other servers delete cgroups correctly. maybe 1 of dockers install-dependencies problem here. docker deleting cgroups correctly.

the fix problem know of, restarting machine.

here output of docker version:

client:  version:      17.03.1-ce  api version:  1.27  go version:   go1.7.5  git commit:   c6d412e  built:        mon mar 27 17:10:36 2017  os/arch:      linux/amd64  server:  version:      17.03.1-ce  api version:  1.27 (minimum version 1.12)  go version:   go1.7.5  git commit:   c6d412e  built:        mon mar 27 17:10:36 2017  os/arch:      linux/amd64  experimental: false

here output of docker info

containers: 3  running: 3  paused: 0  stopped: 0 images: 4576 server version: 17.03.1-ce storage driver: aufs  root dir: /var/lib/docker/aufs  backing filesystem: extfs  dirs: 2359  dirperm1 supported: true logging driver: json-file cgroup driver: cgroupfs plugins:   volume: local  network: bridge host macvlan null overlay swarm: inactive runtimes: runc default runtime: runc init binary: docker-init containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe init version: 949e6fa security options:  apparmor kernel version: 4.2.0-42-generic operating system: ubuntu 14.04.5 lts ostype: linux architecture: x86_64 cpus: 8 total memory: 31.33 gib name: repo id: kt6v:flpi:ehy6:2fsp:nebt:t65w:23ix:irfk:aqm5:kslm:fq3d:57x5 docker root dir: /var/lib/docker debug mode (client): false debug mode (server): false registry: https://index.docker.io/v1/ warning: no swap limit support experimental: false insecure registries:  127.0.0.0/8 live restore enabled: false

the kernel version via uname -a:

linux repo 4.2.0-42-generic #49~14.04.1-ubuntu smp wed jun 29 20:22:11 utc 2016 x86_64 x86_64 x86_64 gnu/linux

cat /proc/cgroups after ~3h uptime:

#subsys_name    hierarchy   num_cgroups enabled cpuset  1   623 1 cpu 2   627 1 cpuacct 3   623 1 blkio   4   623 1 memory  5   627 1 devices 6   623 1 freezer 7   623 1 net_cls 8   623 1 perf_event  9   623 1 net_prio    10  623 1 hugetlb 11  623 1

of these 623 cgroups 616 in /sys/fs/cgroup/memory/user/998.user user 998 git user.

i know not ideal solution removes symptom , not cause, created script can run via cron.

#!/bin/bash set -e cgroup_base_dir="/sys/fs/cgroup" declare -a cgroup_subsystems=('cpuset' 'cpu' 'cpuacct' 'blkio' 'memory' 'devices' 'freezer' 'net_cls' 'perf_event' 'net_prio' 'hugetlb')  cgroup_subsystem in "${cgroup_subsystems[@]}";         cd "${cgroup_base_dir}/${cgroup_subsystem}"         old_cgroup in $(find user/*.user/*.session -type d -ctime +10);                 /usr/bin/cgdelete -r "${cgroup_subsystem}:/${old_cgroup}"         done done

this search session-cgroups older 10 days , deletes them using cgdelete cgroup-tools.


Comments

Post a Comment

Popular posts from this blog

PHP and MySQL WP -

i trying accomplish 2 things here code. writing php directly wp page using insert php plugin. 1) query mysql database , return results 1 column 4 columns. 2) make each result own hyperlink directs user new wp page runs new query on page. i able query show results , turn results dynamic hyperlink, cannot formatting down. right returns result 1 column. here have: [insert_php] $servername = "localhost"; $username = "login"; //edited privacy $password = "password"; //edited privacy $database = "database"; //edited privacy // create connection $conn = new mysqli($servername, $username, $password, $database); // check connection if ($conn->connect_error) { die("connection failed: " . $conn->connect_error); } echo "connected successfully"; //get results database $result = mysqli_query($conn,"select mine mines order mine"); $all_property = array(); //declare array saving property //showing proper...
Read more

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

go - golang pprof for c library code -

i have been trying profile golang application pprof, seems details of c code(linked static library) missed in output result. flame graph from frame graph can see c code time consuming reported within "runtime._externalcode", no details @ all. , sure static library compiled debug enable. so there way make pprof details(like stack trace) of 3rd party c code? or there other golang profiling tool this?
Read more